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ABSTRACT: This dissertation reports the developnient of digital computer 

techniques for detecting changes in scenes fay normalissing and 
comparing pictures which were taken from different camera 
positions and under different conditions of illumination* The 
pictures are first geometrically normalized to a common point 
of view* Then they are photometrically normalized to eliminate 
the differences due to different illuminatioHj camera character- 
istics, and reflectance properties of the scene due to different 
sun and view angles* These pictures are then geometrically 
registered by maximizing the cross correlation between areas in 
them* The final normalized and registered pictures are then 
differenced point by point* 

The geometric normalization techniques require relatively 
accurate geometric models for the camera and the scene, and 
static spatial features must be present in the pictures to allow 
precise geometric alignment using the technique of cross correla- 
tion maxima tion. 

Photometric normalization also requires a relatively accurate 
model for the photometric response of the camera, a reflective 
model for the scene (reflectance as a function of the illumina- 
tion view, and phase angles) and some assumptions about the 
kinds of reflectance changes which are to be detected* 
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These techniques have been ineorparated in a system for 
comparing Mariner 197^ pictures of Mars to detect variable 
surface phenomena as well as color and polarization differ- 
ences. The system has been tested using Mariner 6 and 7 
pictures of Mars . 

Although the techniques described in this dissertation were 
developed for Mars pictures, their use is not limited to 
this application. Various parts of this software package, 
which was developed for interactive use on the time-sharing 
system of the Stanford Artificial Intelligence Laboratory, 
are currently being applied to other scenery. 
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CHAPTER I 

introduction 







The general p|ctur« comparison problem Is to 
geometrically and photometp i oa 1 j y normalize and register 
Images bo that true differences |n the scene cap be 
determlnedi rather than differences In conditions of 
viowingi and to analyze these differences, 

The geometric analysis of photographs belongs to the 
science of analytical photogrammetry (Doyle Cl9663)f which 
generally deals with the tr I anou ! at } on of aerial photographs 
to generate topographic mapsi Most results |n this f|ejd| 
howeveri are approximations which solve suitably linearized 
versions of the equations, Even neglecting this deficiency! 
these techniques are of little use except when extremely 
accurate models exist for the oamera« 

Military aerial reconns I sance probably represents the 
major current application of Picture comparison technology, 
Most of the known techniques fn this field re|y on manually 
operatedi analog Image oorretatorSf and various other analog 


hftfdwftft, Th«r« |s |(tt|t !n tht !lt«rature to daacpibo 
what digital computer taohnlquasi If anyr have botn 
dfvalopad for this purpcoa, 


The system described In this dlssartation Is the only 
system known to scjve the problems of Image normalization 
and registration! 


I, A THE MARINER 1971 MARS MISSION C13 


The primary objectives of the Mariner Mars 1971 Project 
are the observation and mapping of Mars by two orbiting 
spseeoraft beginning ]n November of 1971 and continuing for 
at least 90 days (JPl C19703.), 

An orbiter has the advantage over flyby missions^ such 


Cl] Mariner 8 was launched from Cape Kennedy on May 8, 1971» 
and failed to make Earth orbit, Thereforti tha nominal 
Mariner '71 mission plans must be ohanged. It Is not known 
at this time whether the Variable Features Mlss|on» which Is 
the purpose pf th|s researehr will be attempted, Since most 
of this dissertation was written before May 8* many of 'the 
verb tenses referring to the Mariner '71 Mission should be 


changed to reflect the current situation. 









«s th« Mariner Mtrs 1964 and 1969 mf8S|ona» that 
obaarvatlons from orbit can ba madt over an axtandad parlod 
of time, thue permitting the study of temporal changes on 
the Martian surface. 

The Mariner IV, VI» and VII missions could maMe only 
limited observations of the surface of Mars because they 
were |n the planet^s vicinity for a very short time, To 
accumulate more extensive data |t Is necessary to orbit Mars 
as Planned In the Mariner '71 Mission, or orbit and land as 
Planned In the 1975 Viking mission, 

The two Identical Mariner '71 spacecraft will perform 
separate missions designated as missions A and d. Mission A 
Is primarily devoted to routine mapping, attempting to view 
a large portion of the surface of Mars with the highest 
Dossible resolution, 

Mission B |8 primarily devoted to studying time 
variable features of the Martjan atmosphere and surface, it 
will utilize an orbit which wll) give repeated coverage of 
several different surface areas under essentially constant 
Illumination eng’s, view angle, and spaceoraft altitude 
{Sagan C19693), With such an erb|t, |t Is possible to study 
these areas of the surface to detest changes during the 
90-day mission, 
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The tyisfs 6f ehanoti which «r« txptotcd are elthtr 
trant|*nt or diurntlly or ttaaonally raourrant, Tranatant 
phanomana that hava baan obsarvad on Mara Inoiuda eloudOf 
hazasf and bright spota, Whitf e(euda hava been laen of e|| 
sizas and shapeai from terminator haza lasting a few hours 
v.o danaai 1200-ml|a giants lasting days or waeKs, Yellow 
clouds vary from smaiU danaoi orange or yellow objects 
lasting from one to a f«w days» to objects which start large 
and grow larger until they become a yellow veil covering 
most of Mars and lasting a month or morat These ye||ow 
clouds are almost universally accepted as being dust clouds, 
The Marine? '71 television observations will yield data on 
atmospheric olrcu|atlon by following cloud movement, 

The most obvious seasonal changes on the Martian 
eurface Involve the polar caps and the wave of darkening, 
The surface also exhibits seasonal changes In color and even 
changes )n the slzei shaoei and Internal appearance of the 
various dark areas on the planet, 

The polar caps ar« believed to be deposits of solid 
carbon dioxide condensing during the fall and winter In each 
hemisphere and then subliming during the spring and summer. 
From many biological Points of v|awr the receding polar cap 
Is a local# of great (nterest# and will be observed during 
the Mariner '71 mission, 
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Tht wava of darktnlng Is probably Mars' greatest enigma 
and Is the most dynamic event on the pjanet. It has been 
described aa a progressive decline In the rsfjaetlvlty of 
the darK surface areas Und Increase In contrast with 
surrounding bright areas) starting In local springtime from 
the edge of the vaporizing polar eaoi and moving toward and 
across the equatori Whether the darkening actually occurs 
as a ’'wave'* from the po|e has been argued, This darkening 
might also be accompanied by cpjpp and polarization changes. 
According to the blologloal explanation for these seasonal 
changes, Martian organisms Inhabit th» dark regions, and 
their springtime growth In response to the Increased 
temperatures and humidities Is the oause of the darkening 
events, Several a|ternat|va non-b I o | eg I oa | hypotheses have 
been pfoposed. Including one |n which seasonal changes \'n 
V !nd patterns (due either to meridional circulation or 
dust-devljs) redistribute the particle sizes In the bright 
and darK areas and produce the albed:' changes, 

In selecting scientific objectives for the Mariner '71 
Mission, the study of the wave of darkening was singled out 
for special consideration, since durlns the 1970 to 1980 
deoade, the wave of darkening In the southern hemisphere can 
only be observed |n 1971, The sogthepn hemisphere contains 
most of the permanently dark regions and Is considered the 
more Interesting hemisphere In which to study this 
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Th« two «pio«craft ar« |d«nt|c«! and contain a varlaty 
of aclantlflc Inctrumtnts Inctudlna two tajaylslon camaras* 
an ultravlojat spaetromatarr an infrtrad I ntarfaromatarf and 
an Infrarad radlomatari In tarma of voluma of data 
eoilaetadf tha tajavlaion axparlmant data w|i| outwaigh all 
othar data by savaral ordar* of maonltuda, 

Tha tw'o taiavislon eamaras ar* daaignatad as eafflsras A 
and Tha A caBiara has a focal iangth of 50 nnif giving It 
an 11 by 14 daoraa angular flald of v|awi The B camera has 
a foeal lani.th of 500 mm giving It a 1|1 by 1,4 dagraa field 
of view and hanea 10 times tha resolution, Camera A has a 
oolleotlon of 8 oo|or and polarization filters which can be 
arbitrarily selected, The B camera has a single fixed 
<mlnus blue) color fitter, Tha wave of darKanIng and other 
variable surface features will be studied primarily using A 
eamara oloturas which are taken before perlapsls when the 
sun and opaoeoraft vectors are nearly vertical to the 
surface |n the area being photegrephad, thus minimizing 
uncertainties due to light scattering, Most of these 
Pictures will be taken under an orange color filter which, 
on tha basis of ground obsarvat i ons, should make It easiest 
to detect changes in a I bade, 

Tha television cameras are vldloens, which are 
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shutt«r«d at sptadi ranglns from 6 to 192 ml I ( istconds, The 
Image on the vldloon taraet Is digitized In 42 seconds In a 
format of 9 bits per po|nt» 832 points per linsi and 700 
lines per frame, 0|o|t|ted samples are stored on a digital 
magnetic tape recorder which can hold approximately 36 
Picture* l•ntl| It Is possible to transmit the pictures to 
Earth , 

taoh spacecraft ha»i a radio receiver which |s used to 
control suoh things as propu|s|on» scan platform slewing, 
color filter selection, camera shuttering, and tape recorder 
playback (and transmission), Each spacecraft also contains 
a 10 or 20 watt 2300 MHz radio transmitter which during most 
of the mission w}|| be able to transmit exotrlment and 
spacecraft data to the 210 foot Ooldstone antenna at a 16 
kbps <kl|o»b|ts Per second) rate, At this rate, saeh 
oloture requires about 5 minutes transmission time, A dally 
tape recorder load consists of about 36 pictures, which will 
require a transmission time of 3 hours, This |s about the 
longest period of continuous communication between the 
spacecraft and Coldstone which can b* guaranteed on every 
orbit, IBrjggs C19713), 

To maximize th® scientific return from the Mariner 
mission, It Is necessary to control the picture taking 
sequence to concentrate surface coverage In those surface 
areas which show the greatest variability, Th|s requires 


^ ' 


that pleturas racolyad at Earth be analysed as quIoKly and 
sensitively as possible to detect differences frbm previous 
Pictures of the same areai This dissertation Is the result 
of research In the development of digital computer 
techniques to perform such analysis on pictures from the 
Mariner '71 mission, 

I.B A SOLUTION TO THE MARS PICTURE COMPARISON PROBLEM 

In response to the needs of the Mariner '71 Variable 
Features Teami Image processing techniques have been 
developed by the author at the Stanford University 
Artificial Intel (Iganee Projecti utilizing a. PDP-10 
Interactive time-sharing computer system. These techniques 
have been Integrated Into a system which compares pairs of 
Images taken at different t|meS| from different spacecraft 
positions, and perhaps even different spacecraft, 

Figure 1"1 shows the structure of the system, In this 
figure, the rectangular blocks indicate programs which 
implement the techniques described In this paper, Where 
eppreprlate, the block contains the chapter or section 
number where the technique Is described. Oval blocks 
indicate data (usually pictures}, 

GEOMETRIC NORMALIZATION! S ) noe the Images are different 
perspective views of a spheroids with no surface elevations 
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«ssutn«di ft Qftomfttpic normft ) } zat | on Is rsoulpad to re|ftt« 
oopmon supfftot points. In partlculftPr •■ch Imtfl* Is 
tpftntfopffisd to the sftm« opthoflPftphfc projection of the 
sphsrotd, If there were no errors in our knowledge of the 
spacecraft position and orlentatloni and no geometric 
distortions |n the optical end el.eotronic systems of the TV 
cameras and If our object were a pepfeot spheroid# then the 
two normalized Images should be In exact geometric 
correspondenoe, Chapter II deserlbes geometric 
normal Ization In detal I . 

GEOMETRIC REGISTRATION! Unfortunately# many sources of 
geometric errors exist# with spacecraft orientation 
contributing the largest error. In order to remove these 
geometrio alignment errors# It Is necessary to align 
geometrically corresponding areas (features) |n the two 
Images# A technique was developed which displaces one Image 
relative to the other, searohing for a (dx# dy) translation 
vector which maximizes the cross correlation of the Images 
over a specified area, Maximum cross correlation occurs 
when the Images are properly registered In that area. 

If we know the (dx# dy) translation necessary to 
properly register every point In the gaometr 1 oa | [ y 
normalized Images, then we can analyze the Imagas point by 
point for differences, 
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Tha ft'jssumptlon |$ mad* that tha ml srag | stpat 1 on veeter 
as a agfunctlon of position In the Images Is a smooth 
continuous funotlonf which can be modeled by low order 
polynomials jn two var|ab|aS| This assumption |s realistic 
If the model of the object |a accurate (||S, surface 
elevations ape small or v|aw anojes are similar) and If 
optical and eleetronle distortions ar® smooth and 
Qont I nuous f 

In ppaoticer when we perform a least squares f|t of |qw 
order polynomials In two variables to a set of optimal 
translation vectors on normalized Mariner 6 and 7 Images# we 
get very small residual errors# In particular# when fitting 
1st order polynomials (which have 3 degrees of freedom) to 
from 10 to 100 data points# we usually get residual 
alignment arrors of less than one Picture unit (p|ye|) 
standard deviation, This# at least empIrlcaMy# Indicates 
that the predominant souroes of error cause a smooth and 
continuous misalignment between the two Images, For Maplner 
6 and 7# the known geometric errors are primarily In 
spacecraft (camera) or I entat.l on, which would cause primarily 
a translational# and to a lesser degree# rotational 
ml aa ! I onment between two Images# and therefore 1st order 
polynomials In two variables are a good approximation, 

Having an accurate model for the misalignment between 
the two Images# w® can reoaloujate the orthographic 



projection for one of the Iwigtsi taking Into account the 
itilsal isniTient modeli The resulting Images wIM bo registered 
to the accuracy of the mlaregf atratlon model. Chapter III 
dccorlbes geometric registration In detail. 

PHOTOMETRIC NORMALIZATION! SI.noe the images ape taken 
under different Illumination and view angles, a photometric 
oorfactlon Is required to relate l|oht |ntene|ty levels 
received at the camera to albedo on the surface, If there 
were no errors In our knowledge of the light soatterfng 
function at each location on the p|anet, and no errors In 
tho Photometric response of the v|dIcon, then, In theory, we 
should be able to precisely determine the albedc at each 
point In the Images and perform albedo eomparlsone vO detect 
variable features, Chapter IV describes photometric 
normal liat Ion In data! i , 

PHOTOMETRIC REGISTRATION! Detacting variable features 
necessitates a high degree of photometric acouraoy, since 
some of the variations anticipated are r*iatlve!y small (5X 
or less) albedo changes over rather large areas of the 
planet, These albedo changes may Incraaae the contrast 
between two areae, or may only change the average absolute 
light level In both areas. If the errors and/or noise In 
the calibration of the camera aystam are larger than the 
albedo changes to be detected, than we must Improve the 
calibration using Information In the Images, 
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Thft known r«p«atab|o sourots of photomotrU errors are 
such thinos as vldloon shading (non-unjf orm responsa) and 
residual Images^ both of whjah can be handled in a 
systematic manner C13i The primary aoufoes of error whfoh 
cannot be predicted are an absolute gain erroTi and an 
absolute offset error resembling scattered light, These 
errors, wh^oh may be caused either by errors In camera 
calibration, or by errors |n the light scattering modej for 
the surface, give a system output Y as a function of light 
Input X ast Y » aX + b, 

Using this model for the phetomatrlc misregistration 
function, we can choose areas In the two Images which are 
assumed to be photometrically equivalent and solve for a 
oombined gain and offset error which will minimize the 
difference between the areas using conventional least 
squares techniques, Chapter V describes photometric 


C13 JPL has processed the Mariner '69 Images to reduce these 
errors, Non*unlform vidicen response Is corrected from 
extensive calibration tables for ^he vldlcon. Residual 
Images are the contributions of /previous Images to the 

i 

vidloon output for a given Image (Ije, the vldlcon surface 

/ 

has a memory), Residual Images are partial iy removed using 
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calibration tables and previous Images 
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DIFFERENCE ANALYSIS! One* two ImaBes ar® both 
g«oi7i®tr loa I |y and Photometrically a|lgn«)d^ the analysis of 
dlfferaneei batwtan tho Imasas can begln« Captain classes 
of known differences ape expectedt These Include albedo 
dlfferencee due to errors In the photometrlo mode! of the 
planet, variations of th® photometric function from place to 
Place on the planet, and errors due to the effects of sjopes 
(such as crater rims) on the photometric function, 


Given Images which were taken with approximately the 
same I I |umlnation» view and phase angles, the above sources 
of error should be minimised, The Mariner '71 Mission B 
Images near periaps|s are Intended to satisfy the above 
requirements, and hence these sources of error (except for 
regional variation of the photometric function) can be 
largely Ignorn'i. 


The remaining albedo differences can be attributed to 
temporal variations of the photometric function due to such 
Phenomena as clouds, dust storms, etc, 


Analysis of these albedo variations requires that the 
pixel by Pixel a[bedo differences be reduced to area 
differences and other graphical representations, One useful 
form of area difference display |s a drawing of albedo 
difference contour lines, if the a|bedo difference has a 





weli def|n«el outline thon a drawing of this outline Is 
usefuii Another useful display of dlfferenoes }s a graph of 
the size of an albedo difference area versus time tfor a 
fixed difference level) or a granh of the magnitude of the 
albedo difference versus time Cfor a fixed area). 

Chapter VI describes these difference analysis 
teohn I cues* 

I,C NOTATION 

To faellltate the printing of th(s dissertation by the 
line Printer at the Artificial Intelligence Project# some 
oompromises In mathematloal notation were made, In 
particular# superscripts and subscripts are not possible# 
and many standard mathematical symbols do not exist, 
Consequently# the scholarly use of an abundant collection of 
Greek letters other than «# and e will not be found 
here . 

ZEROJ Zero Is printed with a slash through Its 

zero - 0 

EXPONENTS! Exponents are denoted by the ALGOL 60 
notation, A to the power N |s written asl 

A»N. 
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INDICESI Indloes of vectors sincj matrices are written In 
the ALGOL 60 notation as* 

AC|»J3 # VCkJ 

VECTORSI Vectors which are formed from a collection of 
scalar expressions are surrounded by parentheses: 

Cu»vJ $ <x/y/z> 

When It Is necessary to specify a column vector} the 
transpose notation Is ussd> 

CX}y»z)' 

matrices: Matrices which are formed from a collection 
of scalars expressions are written using as many lines as 
there are rows In the matrIXi ThuS} the 3x3 Identity matrix 
would be written: 

10 0 
Is 0 10 
0 0 1 

The transpose of a matrix M Is written* 

transpose CM) » M# 

The Inverse of matrix M Is written* 




tf 


INNER PRODUCTJ The Innar product of two vectorH VI and V2 Is 
wr ] tteni 

<V1,V2> 

EXPRESSIONS* Arithmetic operations between 8ca|apa, 
vectors and matrices are denoted In conventional 
mathematical notation whenever poaslblei Consequent (yi a 
particular symbol* auoh as *» can mean different operations 
depending upon the context* 

EUCLIDEAN NORM; The |enath of a vector V (Euclidean 
norm) Is written* 

IIVIl 

INTEGER FUNCTION* The Integer part of a real number X 
(|*e* the greatest Integer not exceeding X) Is written* 

m 

SUMMATION* The sum of an expression over an Index 
variable Is written using two llnasi the first containing 
the word ”sum” followed by the expression* and the second 
line containing the name and range of the Index of summation 
under the word ”sum», Using this notation* the product of 
two matrices A and B would be written! 

CCI,J3 » sum ACI»k]#BCk* J3 
ISkSn 

INTEGRATION! The Integral of an expression Is written 
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analogously to a sum, The Intaaral of function fot) ov«r 
th* range |s written! 

Integra! f(x> 

-lS«Sl 


I,D vocabuuary 

Some of the words used |.n this dissertation are 
unoommonr eome are local Ismsi and some are highly teohnlsaL 
Thersferei eome such words are defined! 

albedo - The ratio of the reflected light to the Incident 
light normal to a surfaoet A white Lambertian 
surface^ normal to the Incident sunltghti would have 
an albedo of onsi 

apoapsis - apogee - The point In the orbit moat distant from 
the planeti 

periapsls - perigee- The point In the orbit closest to the 
p lanete 

pixel “ Abbreviation for *'pIotupe e!emant*»i referring to the 
light value at a point |n a picture, 

The next four chapters describe In detail the 
teohniQues for normalizing and registering Images, 


CHAPTER U 




I: 





GEOMETRIC image NORMAUI2ATION BY "DEAD RECKONING" 

Imege norma 1 1 zRt 1 on by »daad pscNonlng" refars to 
tachnlauas which Qeom«tr lea I |y and ohotomfitr I oa I I y reglatap 
imaflos using only calibration Information suoh as oamepa 
oosttlon and ortontatloni oamera sensitivity, etc, rather 
than Information contained within ths Images themselves, 
The quality of such normalization techniques Is determlnid 
by the quality of the calibration Information, For Mariner 
6 and 7, errors In the calibration result |n a geometric 
misregistration of 5 to 10 pixels (25 to 50 kilometers on 
the surface), 

problem* Given two Images taken from different 
epaoeeraft (camera) positions and or I entat I cnsi possibly 
from different spacecraft (camepaa)i geometrically transform 
the region |n each Image which is common In the scene (on 
the surface of the planet) so that the two Images can be 
Qompared pixel by pixel, 
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definition* An imsfla « r§tj function of two vnrUbjts 
FCu»v) peprcson*'' llflht Intensity p®celvcd nt point 
(u»v) on the |cai,i*t Q}<p.ris c? some Imaalng davloe (eci a 
camera) , 

All of thf 'rtifist scanning aystams of Intepasti howavepi 
are dlsoreta ay^i^tams which quantize the light level at a 
fixed number (eg, 312) of Intensity levels over a fixed 
rectangular array of points (eg, 945 x 702). 

Such discrete Imaging syatana# furtharmope, encode the 
light level Integrated over an area rather than at a point, 
This Integration can be formalized asi 

F(x»y) s Integral ( f(uiv) * gCx-ujy-v)) 

"•*<U, v<+« 

where f Is the Intensity function at a point and g Is the 
’•point spread" function of the Imaging device (Rosenfe|d 
C19693,p 44), which Is usually adjusted by optical or 
electronic defoousing to minimize the errors due to aliasing 
Introduced by discrete sampling, - 

In terms of scene ceordlnatesi there exist two 
oeometPic tpanef ormat Iona Ti and T2 which map coordinates of 
oolnts In the images II and 12 Into scene coordinates* such 
that when (ulivl) and Cu2#v2) represent the same point 
(x*y*z) In the scene, the following Is trust 
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Tl<ul»vD « UiYiZ) = T2(u2iv2) (ffiD 

Tht n«xt sactlons dsrivt tpfinaf ormat 1 ons T1 and T2 
using fltometrle medals for tha aoans and oamepai 

II, A GEOMETRIC MODELS 

PROJECTIONS OF SCENES! A ProJagtlon of a sotna Is some 
goomatrlo maoplno T(x»y,s) which maps tha eoordlnatas of 
points in tha 3*D soena Into coordinates of the 2-D 
projactad Image, Soma points In the aoene do not mao to the 
projaotad Image («,g. points on tha opposite side of an 
objaot being viewed by the eya9, 

Tha most fam|||ar projeet|on Is the perspective 
projection (F|g, 2-la), which maos the point Ps In tha 3-D 

scene Into tha point PI which Is tha Interseotlon of the 
Image plane and the straight line through the |ans center 0 
and Ps, If there Is any other point Pa on the line between 
0 and Pa, than Ps |s said to ba *<oco[uded" by Pq, Occluded 
points are not mapped to tha Image plane, 

Another useful projection Is the orthographic 
projection (Fig, 2»lb) whioh maps the point Pa In the 3-D 
scene Into tha point PI such that Ps Is on tha line nermal 
to the Imaga plans at P| , 

CAMERA MODEL! The osmera can be geometrically modeled 
as a Isns center and an Image plant (Fig, 2-la}, At each 
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point (uiv) 

on the 

Image 

Planer 

the 

camera records 

the 

amount of 

1 iQht 

received 

from 

the 

scene along the 

1 Ine 

through the 

lens center and 

point 

Su» v> 

1 Thusi the camera 


Is said to generats a parspiotlva Image of the soane, in 
order to normalize Images taken from different camera 
positional It Is necessary to determine the soene 
coordinates of points In the pepepeotlva Imasesi 

SCENE MOOEUSt Theoret I ca | lyi general soenes are easy to 
mode! geometp I oa | |yi but In practlos this oan be very 
difficult to accomplish, A very general class of scenee may 
be modeled by assuming that the scene Is made from objects 
whose aupfaoee are onactue and ape described by a function 
H(i(iyiZ)a0, A scene consisting of a sphere of radius r 
centered at the origin can thus be modelodJ 

H(x»yiZ) « Xt2 + ytZ + z»2 - ptS s 0 (2,2) 

The Planet Mars can be approximated by an oblate spheroid 
with equatorial radius Req«3393,4 Km (equator In the x-y 
Planer with zs0) and Polar radius Rpo|b3375,6 km (poles 
along the 2®axi3), with this modeli H becomes) 

H(x#y»z) « x»2 + y<'2 * ( z*Roq/Rpo | ) t2 - Raqr2, (2,3) 
II, B THE INVERSE PERSPECTIVE PROJECTION 

Inverse projections map coordinates of points In Images 
which are projections of soenest back Into coordinates In 
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tht scenoi Geometric modciv for the eoene and the 

projection are required In order to Invert the projectlont 

The Inverse perspective Image projection Is the 

projection from two dimensional perspective Image 
coordinates to three dimensional spatial coordinates, This 
projection maps Point (u»v) In the perspective Image plane 
to the oiosest point (x,y,j;) In the 3»D scene which Is along 
the line through the lens center and <u»v), Any point along 
that line la given |n camera relative coordinates (the lens 
center la the orlglm with axes U|V| and w) as a function of 
the depth parameter d as d«(uiv,f}-f where f (a the Image 
distance of the oamera (F|g, 2-lo), 

To transform camera relative oeordlnatee Into scene 
relative coordinates, a linear transformation u |s doflnedi 

L(u,v,w)) 3 R(urV,w}' + Pc <2, 4) 

where! 

1) R Is a rotation matrix from camera space orientation to 

scene orientation, Appendix A d'^*-ivv6 matrix R from 
geometric calibration data, 

2) Pc Is the position of the oamera f3>-?tlve to the 

origin of the scene coordinate system, 

Using these definitions# the Inverse perspeetlve 
projeetlon T of point (u#v) |a defined In scene coordinates 
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by finding the unknown depth partmeter d such that? 


T(u,v) ■ U(d#(UrVif)') (2,5a) 

* R<d#(ui v» f ) ' ) + Po (2.5b) 

a d«R(urv»f)' + Po (2,5o) 

such that d>0» d»mln# and (2,6 ) 

0 a H ( T(uiv) ) (2,7a) 

« H ( d*R(u»Vrf)' + Pc ) (2,7b) 


where the ppimad vectors denote column veotops, Equation 
C2,5) requires that T(UfV) be a perspeot|ve view of the 
scene from a camopa at position Po with orientation defined 
by R and with Imape distance f, Equation (2,7a) requires 
that T(u,v) be a point In the scenes and (2,6) requires that 
it be the point closest to the camera, 

Using the above dafinitlonsi the perspBctlve projection 
T' Is the Inverse of T and can be defined as follows! 

Let the transformation T’' be defined* 


?"(x,y, 

Z) s |nv(R)»C (xjVi Z) '-Po) 

(2,8) 

where InvCR) Is 

the Inverse of matrix 

R, Note the 

f 0 1 lowing) 



T”(T(u,V) ) 

= T«(R(d*(UiV,f )' ) + Pc) 

(2,9a) 


« !nv(R)*(R(d#(u,Vif)' ) ~ 

Po * Pc) (2,9b) 


= lnv(R)*(R(d*(u, v,f ) ' ) ) 

(2,9c) 


- d*(uiv»f)' 

(2,9d) 
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(u»v»w)' « T«<x#yiJ) 




th«ni the perspective projection T' Is dafinsd asl 

C uj V) 

T'lxiVi*) of# <2, 3.0b) 

w 

Note that w«d#fi 

U,C SPECIAL CASES OF THE INVERSE PERSPECTIVE PROJECTION 

Eauatlons t2,5<.2,7> are In genera! vary difficult to 
solve because of the complexity of H, This section derives 
their solution for the speeU! oases of spherical and 
spheroidal soane mode|s» 

SOLUTION FOR A SPHERE! The Inverse perspective 
transformation problem can be easily solved fop known camera 
position and orientation with respect to a spherei In 
partlcuiari for a sphere of radius p centered at the aoans 
origin^ we have for a given point P on the sphere* 

1]P|1 s r, |.8t Pxt2+Pytg+Pzt2 K pt2, C2,ll) 

So» given th» parspectjve projection of P In Image 
coordinates {u»v)» w® know thati 
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0 = I !T(Ui v) 1 1 1-2 - pfZ t2,12a) 

* I IRCd*(u»v»f J' )+Pel I ^2 - rt2 (2,3.2b) 
■ I |R| I |d«(uiVf f ) ' I 1 *2 (2,12 o> 

* 2*<R(d»(u»Virf )' ) »Pe> 

I IPol h2 « r*2 

B dt2 # j I (u»v,f ) I !»2 (2,l2d) 


+ 2*d * <R(u»v,f)'»Pe> 

* I |Pcl 1*2 « pt2 

d Is found by solving th« qusdrstlo squatloni 

d » (-b - sqpt(b?2"4#a*c) )/(2*a) (2,13) 

whops 

a = I I tu»v»f ) I l»2 3 ut2+v*2+f*2 <2, 14a) 

b « 2»<R{u»V»f )'»Po> (2,14b) 

0 s I |Pc! I *2-P*2 (2,14c) 

Ths smaller solution (- aqpt) of the quadratic equation Is 
the only msaningful one s|nos ths oolnt on the sphere 
corresponding to the larger solution Is at a greater 
distance fr-om the camera and Is therefore oaoluded by the 
point oerrsepondlng to the smaller solution, Complex 
solutions oorrespond to camera rays which do not Intersect 
the sphere, 

SOLUTION FOR AN SPHERDIDJ The solution fop a sphere 
easily generalizes to a spheroid by Introducing an 



«oc«ntpio!ty constant a along tha Z"ax|a (North to South 
pels a^tls} such thatt 

XT2 + yts + » r’2 (2il5) 

If the Bocantrlclty matrix E is defined as* 

10 0 

E 3 0 1 0 <2.16} 

0 0 • 

then d Is found by solving tha following quadratic equation* 

0 s I |E*T<Uf v) I I ?2 » r^2 (2,17a) 

a I |E*(R(de(u«V>f))'^Pc| |t2 - r»2 <2, 17b) 

B d»2 * I |E*R(u»Vif ) I h2 (2,170 

•**2*d*<E#R(u# Vf f ) »E(Pe)> 

+1 |E(Po) I|t2 - pt2 

U,D EXAMPLES OF GEOMETRIC NORMALIZATION 

This section contains examples of the geometric 
normalization techniques applied to f ar-encountar Mariner 7 
pictures of Maps, These pictures were taken when the 
spacecraft was sufficiently tap from Mars to see the entire 
disk of the Planet, 

Figure 2"3 shows the disk of the planet (the clro|e) 
with the north pole at the too as seen from an orthographic 
projection In the direction B latitude, 0 longitude (l,e. 
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<0 lat,, 0 long*) |$ th« aentor of th* olrcle)* The dottod 
||n«s surrounding the olrole Indicate the outlines of two 
tlotures which ape designated 7F75 and 7F78, The vectors ]n 
the circle indicate the direction to the spacecraft and the 
sun from the oentrai points of the two pictures* The square 
indloates the area on the surface for which an orthographic 
projeotlpn of each picture will be generated* 

Figure 2-4a shows the area of the picture 7F75 (the 
dotted area) which fs speolfled by the square In figure 
The rectangle 1s the 945x702 pixel outline of picture 7F75, 
Similarly# figure 2-4b shows the area of 7F73 which Is to ba 
orthographical ly projected* 

Figures 2-5a and 2-5b show the pictures 7F75 and 7F78, 
as seen from the spacecraft, Note that the crater (Nix 
0|ymp|ea)# which Is in the upper mldd|« part of 7F75* Is In 
the upper right part of 7F78, This Is because the Planet 
rotated by 34 degrees between the two olotupes* A scale 
factor difference Is also obvious, 

Figures 2-6a and 2-6b show the orthographic projection 
of each Image as specified by figures 2-3* 2-4a* and 2-4b, 
In these normalized pictures* the crater looks about the 
same, and various features match* However* there Is a 
definite geometric error In the registration of these 
Images, 




CHAPTER III 




accurate GEOHETRIC registraticn of normalised images 
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in, A FORMAL OEFINITION OF GEOMETRIC MISREGISTRATION 

Let U8 suppose thet we have two gsometr I on 1 1 y 
normalized Images (F1,T1) and <F2|T2) where the T{ are 
transformations from Image eoordtnates to actual S-spaoe 
coorolneteesi rather than those predicted by Chapter 11, 
which contain errors due to the camera and scene models, 
The FI are the Intensity funotlons in the Images, The 
Images are said to be "geometr I oa | I y m t sreg I stared" at point 
(u,v) If> 

Ti(u,v) ^ TZtUivJ (3,1) 

We will define two functions dutuiv) and dv(uiv) which 
represent the misalignment of two Images In the u and v 
directions respectively as a funotlon of the position In the 
Images asi 


(du(Uf v) f dv(u# V) ) ■ ((duidv) iTl(U»v)»T2Cu+duiV'i'dv) J (3,2) 
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2"5a 

7F75 Original 


Figure 2«6a 
7F75 Normalized 


Figure 2"6b 
7F70 Norma I I zed 


Figure 2-5b 
7F78 Original 
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Th«ae m|aal lonmanl; functions t*|l us that» fllv»n a point 
<u»v) In Imaos 1, this sams point In the soene u at 
(u+duiv+dv) In Itnnat 2| 

If we have the misalignment functions defined at every 
point In the discrete ImageSf then we can define a new Image 
(F3»T3) which Is exactly registered with Image 1 as follows: 

F3(u»v) *s F2(u+du(uiV)#v+dv(u» v) ) <3, 3) 

(assuming an adequate model for F£ between data points) 

T3(uiV) * T2(u+du(u> v) I v-^-ovt u» v) ) (3j4) 

* TKurv) 

The next sections empirically derive the miaalignmant 
functions for a pair of Imagesi 

ni.B MODEL FOR MISREGISTRATION 

Because of geometric errors primarily In tha Mariner 
'69 camera model (camera position and orientation), Images 
which have been geometrically normalized by "dead reokonlnC' 
are not exactly registered, It can be shown that small 
errors |n the camera mode! causa primarily a translational 
misalignment between geometrically normalized Images, There 
are also small rotational and scale factor errors, Higher 
order erropa might exist due to residua! distortions In the 
optics and errors In the 3»D model for the scene. 
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On* might obs*rv* that olv«n two lm*g*9 which er* 


aporex I m*t* I y aligned, peoDi* very eaal|y match moat 
ffaturaa (such as oratars) which correspond. However, the 


mechanisms used |n Image matching by people are not 


neoessarlly those beat for automatic Image alignment by 


eomputer 


The predominantly translational nature of the 


misalignment of Images suggests that, when local areas In 


Images are appropriately translated, the Images will match. 
If the two Images were taken under sufficiently similar 
photometric conditions (such as Illumination, color filters. 


etc,) then the Quality of the match can be measured by 


statlstloal cross correlation 


The registration technique 



which has been developed Is to maximize the local cross 
correlation of the Images as a function of the translation 
of one Image with respect to th* other, 

An alternative procedure would be to search one Image 
for a distinct feature such as a crater or some other 
topographic feature, and then search tne second Image In a 
predicted area for th* same feature, Such feature 
recognition, however, reaulres that eaoh Image which Is to 
be aligned contain the necessary number (and Quality) of the 
types of features which can be recognizee, Experience with 
a crater finder Cl3 shows that |n order to reliably locate 
craters, the contrast of the crater r|ms must be good, and 
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thmt th» aignt l-to-no ls» patio must be hlehi Even when 
thsse oonditlons are met» the cross ooppeiatlon alignment 
procedure seems to be supertor; because of Its relative 
Insensitivity to different types of terrain and to noise, 
There also appears to be no Inherent speed advantage fop a 
feature deteotor, One outstanding virtue of a oratep finder 
would be the ability to locate craters and develop a 3«0 
depth model so that the photometrlo errors due to crater 
slopes could be removed, Such slopes, however, cause 
significant errors only when the Illumination and viewing 


angles are 

quite different between 

the 

p f otures 

be I ng 

peg 1 steped. 






Cross 

eorrelatlon 

maximization 

was 

se 1 acted 

for 

geomatr I o 

a 1 1 gnment, 

beoausQ of 

I ts 

simp! 1 0 1 ty 

and 

applicability to varying 

terralne, 





C13 A erater finder was developed which looks for high 
contrast edges which contain large circular arcs. Many 
craters are recognized by the fraction of their p|m which Is 
a high contrast crescent. The outside curve of the cresoent 
Is a olroular arc wfth approximately the same radius and 
center as the orater. 
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Ul.c CROSS correlation as a measure of the quality of 

REGISTRATION 

CorraUtlon Is a statlsticai niBRSupa of agpaament 
betwten two functions, For two random vaplabjas X and Y 
def|n«d on a dIscpBte set of n points, thelp normalized 
oross correlation tcopJ la defined |n terms of the means (E) 
and standard deviations (sd) ast 

E(XeY) « E(X)»E(Y) 

oor(X,Y) B (3,5) 

8d(X) e ad(Y) 

where 

E(X> " sum (XC|3> / n (3,6) 

isisn 
and 

Sd(X) * sort ( E(XtZ) - E(X)tg ) {3,7) 

To apply cross correlation to an area of two Imagas 
with Intensity functions FI and F2, the one dimensional sum 
In eouatlcn (3,6) becomes two dimensional# and W« make the 
followlno definitions for the variables x and Y, Given 
(Uiv) as the center of a (Zn+l)x(2n+l) area to be correlated 
and a (du#dv) translation veoton define X and Y asi 

X[|,J3 B Fl( l+u-n»J+v-nJ- 0£l»JS2n (3,8a) 

YII!»J3 * F2( l+u"n+du» J+y-n+dv) (3,8b) 

The normalized correlation function has the property 
that It does not vary wjth the size of the correlation area# 
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or with th» w«ans and atandard diV|at|ons of X and Y, 
SufaJaot I vt ly thia auaranteaa that th« correlation function 
Is not affeoted by the average lntena|ty |eve| and level of 
oontraat In the windowi The quality of the correlation 
maasupa Isi however# strongly related to si gna I ■*tO’’no j sa 
ratloi Inoreaaed noise w||| Increase the standard 
deviations In equation (3|5) without Increasing the product 
term E:(X+JY)t 

This effect can be derived by assuming that the two 
windows are Identical except for unoorrelated noise, Then 
they can be represented* 

X«H+Nl#YaZ+Na C3,9a) 

Where 2 Is the «true" value of window, If Ni and N2 are 
unoorrelated noise, and have the same standard deviations, 
then, the cross correlation of X and Y |si 

E((Z+Ni)*(2+N2) ) - E(Z+N1)*E(2+N2) 


eor(X,Y) « — — — - (3,9b) 

sd(2+Nl> » sd{2+N2) 

E(H#Z) - £(2^*E(2) 

« (3,90) 

sd(H+Nl) * sd(Z+N2) 

5* var(Z)/var U+N) (3,9d) 

* 1 - vap(N)/vap(H+N> (3,9a) 
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where var Is the vaplaries. High unoopralated noise levels 
oonsaquently will decrease the value of the copraUtlon 


functloHt This suggests that the correlation function 
shoulQ’ bo evaluated only (n areas where the standard 
deviations of the windows are s|gnlf loant |y greater than the 
standard deviation of the noise, and that the search for 
maxlipum correlation should fjnd a correlation value 
approx Innate |y equal to the value of equation (3,9e)i 

Tor a desired level of correlation quality* the window 
size oan be adjusted aeoord|ng to the local standard 
deviations of the two tnnages, Increasing the window size 
(lie. Increasing n) excessively will cause poor results due 
to non-trans I at I ona I (rotation and scale factor) distortions 
between the two Images (Appendix B), 

iii,D local cross correlation maxihihation to determine 

LOCAL MISREGISTRATION 

Analyses of the errors In the Mariner 6 and 7 geometric 
model show that Images which are normalized by "dead 
reckoning" should contain prlmarl|-y translational errors In 
alignment with smaller rotational and scale factor errors, 
Tha results (n tables and figures 3-1 to 3-7 confirm this 
orealctlon* and show that constant transjatlonal errors 
dominate other errors by a factor of more than 10*1, The 
PMsenoe of non-trans 1 at ! ona | errors puts an upper limit on 
tha size of the (2n+i )x(2n+l) correlation wIndoWi 
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TharafopBi to align two Images using oorpel*tlon» It ts 
sufficient to find the (duidv> translation of one Image with 
respect to the other which maximizes the value of the 
correlation function around each point (utv) In the Images, 
This (du»dv> translation Is called the ’'misalignment vector” 
for the area of corraiatlon centered at (u#v), It has not 
been necessary to maximize the correlation with respect to 
other parameters sych as rotation and scale factor changes, 
because the magnitudes of these errors are small relativa to 
translational errors# and It Is possible to find the local 
correlation maximum using only translations, These higher 
order errors are found when many different misalignment 
vectors are modeled over the entire Image as described In 
1 1 1 1 E I 

search for maximum - STRATEGIES* The subject of this 
section Is to describe two different strategies to search 
for the translation vector which Produces the maximum 
oorrelatlon between two Images, The predominant constant 
translational misalignment suggests that the search for 
maximum correlation should be confined to the neighborhood 
of that predominant error vector, 

Tha primary reason for considering these search 
strategies Is to reduce the number of evaluations of the 
correlation function and thus Improve the oerformanoe of the 
Image registration system. 



Th« Rllgrunfint strategies aroi 

1) Global strategy* det«rm|na the predominant global 

misalignment of the two Imagest 

2) Local strategy: determine the local misalignments of the 

two Images by searching In limited neighborhoods 
predicted by the globsl misalignment model, 

GLOBAL SEARCH STRATEGY! The purpose of the global 
search strategy Is to search for the translation vector 
which Produces the maKimum correlation between two Images 
without knowledge to limit the search to a small 

neighborhood, Because the correlation function Is evaluated 
only at Integer values of the <du,dv) displaoement vectori 
the global search strategy oould be Implemented by 

exhaustive evaluation of the correlation function at all 
Integer translations, Howeven some analysis of the 

correlation function, and the areas of the Images being 
oorrelated, shows that one oan ijmit the number of 

evaluations of the correlation function considerably, 

If one knows the sharoness of the correlation peak and 
Its amplitude relative to other relative peaks, then one can 
determine a search grid spao I wh I oh will guarantee finding 
the absolute peak, If fop Instance, the absolute peak Is k 
units wide at the level of any other relative peak, then the 
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global saaroh can be carried out on a kxk gridi reducing the 
number of evaluations of the correlation function by a 
factor of k*2 (see figure 3-0a>i Various parameters (such 
as window slzsi spatial averaging* etct) can be adjusted to 
broaden the peak sufficiently to allow flexible choice of 
the grid spacing tF'losi 3»0o and 3-0ci show th® effects of 
spatial filtering on correlation)! 

If one wants to search for tbe correlation maximum by 
evaluating the correlation function onjy at points on a kxk 
pixel grid* then the sampling theorem says that one should 
limit the frequency spectrum of the correlation function to 
spatial wavelengths longer than Sk pixels, This |s 
successful only If the power spectra of the two Images have 
sufficient power at spatial wavelengths longer than 2k 
olxtls, Tha disadvantage of low pass spatial filtering Is 
that high spatial frequenoy Information^ which usually best 
character I zee topographic faaturesi Is lost, Therefore* a 
search for the local correlation maximum of unfiltered 
Images is desirable In a limited neighborhood around the low 
frequency maximum, 

LOCAL SEARCH STRATEGY* When the predominant 
translational misalignment between two Images Is known, the 
search for the local misalignment can be limited to a small 
neighborhood* thus limiting the number of evaluations of the 
correlation function. The local search strategy begins 
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ttarohlna for maximum oorrolatlon storting with a 
trtnsistlon vsotor which Is orsdictsd oy a ml srsg I strat I on 
modsi bassd on ths orsdomlnant global m I srsg I strat I on vsetor 
and prsvious local rsglstratlon vsotorSi Ths strategy 
ssarehst a llmitsd distanes In 8 different dirsotlons 
looking for a maximum, A h|||*ol|mb search then follows* 
until a (du*dv) point Is found suoh that all 8 neighbors 
give a lower correlation va|uei 

modeling the correlation function at non-integral 

TRANSLATIONS! Given ths valuss of ths correlation function 
at a dlsersts sat of Integer translations* It is useful to 
model the function at non-integral translations using 
interpolation or* eou I va I snt I y * fitting linear oomblnatlons 
of functions (such as polynomials^ to the data. 

In parttoular* one can f|t a polynomial In 2 variables 
to the correlation function at Known translations* and solve 
for the maximum of the polynomial. The method chosen was tc 
use a least souares f|t of an Nth order polynomial In 2 
variables* and solve for Its maximum using a 2-d I mtns | one I 
generalization of Newton's method (KowaMk [19683 pp, 
65-71) In the vlc|n|ty of the maximum empirical value of the 
correlation function, 

Although It Is difficult to prove that the correlation 
function can best be approximated |n this fashion, there Is 
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Eirplrlcal evfdencei Inoi*«aalng Ni tha order of the 
polynomials f]t to tha oorralatton surfaeOf produeas 
quintuples (uC |3ivC IJrduC n,»dvCI3»oC ID) which are 
better modeled by section III,E In terms of RMS 
error (the oCID are the values of tha correlation 
funetlon which are used as weighting coefficients 
for the least squares f|t In seotlon IIltE). 

2) Theoretical evidences The correlation function tends to 
be smooth and radially symmetric about Its maximum 
value (see figs, 3-0a to 3-0d), From the Fourier 
analysis point of view* the Fourier transform of the 
oorrela.lon surface Is the product of the Fourier 
transforms of the two Images (complex conjugate of 
one of them}, If the two Images correlate we||i 
then the correlation surface Is approximately 
equivalent to the autocorrelation surfaces of elthsr 
cf the Imagasf which tend to be radially symmetric, 
If both Images have most of their spectral power at 
low spatial frequenclesi then the correlation 
surface will also have most of Its spectral pov^er at 
low spatial frequenolesj and the correlation surface 
will be oor respond I no I y smooth and broadly peaked 
around the nominal misregistration vector, 
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EXPLANATION OF FIGURES 3-0a,biCtti« These figures show 
windows from pairs of glcturosi and cross saotlonv of the 
oorrelatlon surface In tha v|oIn(ty of the correlation 
maximum, Thesa cross sections are graphs of the value of 
the correlation function with one of the variables dx and dy 
changing and the other constant, 

Figure 3-0a shows the eorrelatlon function for a window 
containing the crater N|x 0|ympIoa. The crude 30x30 pixel 
windows are data from the pictures 7F75 and 7F79, The first 
graph Is th* correlation function with dx changing and dy = 
8, The second graPh shows the correlation function with dy 
changing and dx «= 35, The correlation surface Is modeled by 
fitting a second order polynomial In dx and dy to Points 
around the correlation peaK (dx « 35p dy *• The maximum 
point on this surface Is found to be (dx = 6,08/ dy = 35,36) 
attaining a correlation value of ,75» using a 
two-dimensional Newton's method, 





Figure 3-0b shows an area which does not contain any 
particular features, The correlation function In this area 
Is also smooth and symmetric. 

Figures 3-0c and 3-0d show the bshaviop of the 
correlation function when low fpeauency Information Is 
removed fpom a picture (these areas are from different 
Mariner 6 and 7 pictures), Figure 3-0c la the correlation 



function for an area containing savtral crataps, Figure 
3**0a shows the oorpejatlon function for the same area after 
low SDQtlal frequencies have been removed (subtracting the 
local average Intensity computed over a 10x10 area from each 
point), Removing low spatial frequencies clearly reduces 
the width of the correlation peak# 

III.E MODELING THE MISREGISTRATION AS A FUNCTION OF POSITION 
IN THE IMAGE 

There are several alternative methods available for 
modeling the misalignment functlonsi One method Is Mth 
order Interpolation (usually M*2) In 2 variables in foeel 
regions of the Images, Another method (which was chosen) Is 
to fit polynomials In 2 variables to the entire set of 
misalignment vectors, and then minimize the mean squared 
error between the polynomials and the empirical data points 
(the method of least squares), Ifiterpo | at I on Is good If the 
errors In the empirical data points are small (that Is, the 
correlation search always works well), and If there are 
enough data points to adequately cover the entire Image, 
Polynomial fits to the entire images were chosen because 
they have good smoothing properties on noisy data. 

The misregistration functions du(u,v) and dv(uiv) are 
empirically derived from n quintuples of the form 
(UC 13, VC UidunDfdvC n,ci:i3), where <duCU,dvCl3) Is the 


4ran8l«tion viciior whloh ma«lmUe8 the corpalatlon oCl3 
between the 2 Images over a wlntjew centered at <uCI3«vi;i;n, 
Por aliPD!Ioity» these functions were chosen to be Nth order 
polynomials In 2 variables of the formi 

du(UfV) » sum ACI I J3*u+ l*v* J t3il0) 

0S|+J<N 

Tho iradltlonai least squares approach Is to ohoose du(u,v) 
such that! 

sum ! dut uEk3 ) vCkJ ) -duCK3 >*2 » mln 
k 

s sum ( sum AC 1 1 J3*uCk3» l♦vCk3'^ J ” duCk3)*S (3,12) 
k UJ 

This least squares problem degenerates to the solution of an 
(N+D*<N+2)/'2 order system of linear equations, The 
polynomial fop dv Is construotad similarly, 

The RMS error of the polynomial fit to the empirical 
data points Is a good measure of whether the order N of the 
polynomials Is high enough, and 1 fl |n fact polynomials are 
the appropriate functions to f|t to the data, To account 
for tho number of degpaea of freedom |n the polynomial fit, 
the following measure Is used! 

Cdu(uCk3,vCk3)-duCk3)»2 

sqrt C sum ----- ) t3,i3) 

k Nk - (N+l)*(N+2)/2 

where Nk Is the number of data points, 
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since the empirical data points represent a sampllntj of 
a surfaosi we would like t.ie polynomials to mode! that 
surface everywhere as well as sample points, A useful test 
Is to fit polynomials to one set of sample pointsi and than 
measure the RMS error between these polynomials and another 
sampling gf the surface, 

The above least squares b!-var|ate polynomial 
approximations haVe been used successfully tomode| the 
misregistration vectors between Images, Tables 3">1 to 
and figures 3-i to 3-7 show some actual mlsrec I strat ! on 
veotors and the results of fitting polynomials of various 
orders. 

The tables contain the misalignment vectors (DU;DV) 
which maximize the correlation funotlon over a 21x21 pixel 
window centered at points (U#V>, COR Is the maximum value 
of the correlation function, The residual error vector 
{DUerr»DVerr ) Is the difference between the empirical 
(DU,DV) vector and the vector predicted by the polynomial 
model, ||error|| Is the Euclidean norm of the residual 
error vector. 

The figures graphically show the data In the 
oor PBSDoncH ns tables. The square on the |aft represents the 
entire area of the 200x200 pixel pictures, The vectors In 
the square start at points (U#V) and have length and 
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sHr^ctlon U0*OU«rr,10*DVepr ) , The sjngje digit at {U»V) 
peppesents the vajue of the cori*B|atlon maximum (l»Oi 20% 
to 29X (8 represented by 2) t The circle represents residual 
eppopa of on® standard devlatloni The endpoints of the 
pesidual vectors (Ouep p * DV« r r ) are plotted relative to the 
center of the circle. 

Tables and figures 3-1 to 3-3 show residual errors 
before bad m I sres | atr at | on vsotors have been eliminated by 
'•blunder removal” (explained |ater)i Table and f.lgure 3-1 
show the residual errors from zero order polynomial fits to 
iA data points, The predominant translational error Is 
DU«34 and 0V»9 pixels. The weighted RMS error Is 5,25 
Pixels, where data points are weighted according to their 
correlation value COR, The corrected error of equation 
(3,13) Is 5,65 pixels, To oompare the quality of the fit, 
these corrected errors will be used, Table and figure 3-2 
show the residual errors from first order polynomial fits to 
the same data points, The oorrested error, which is 4,47 
pixels, has not been reduced much, Table and flgur® 3-3 
show the residual errors from a second order fit, The 
corrected error Is 4,25 pixels* which Is not a significant 
reduct I on , 

Tables and flgyres 3-4 and 3-5 show residual errors 
after 4 ''blunders” have been removed, Blunders are 


misregistration vectors whioh do not fit any systematic 



mods?! These vectors result from cross correlating areas 
where these Is some form of systematic difference between 
the pictures such as cloudsi or crater “shadows" which have 
moved due to changes In I | | umlnatloni These blunders are 
removed by finding residuals <DUerr» OVerr) which are large, 
and do not eluster with the other residuals, 

Table and figure 3-4 show the pesidual errors after a 
first order fit, The corrected error |s now 1,03 p|}<e|s, 
The second order fit of table and figure 3-b has a higher 
corrected error of 1,18 pixels. 

Figures 3-6 and 3-7 show the res|dua| errors from first 
and second order Polynomial fits to a much larger (about 60 
points) set of misalignment veotofSi after blunder removal. 
The first order fit has a coprectad error of 1,26 pixels, 
which Is fa|p|y consistent with figure 3-4, The second 
order fit has a corrected error of 1,02 Pixels, which Is 
somewhat better than the first order fit, 
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TABLE 3"i 

POLYNOMIALS FIT TO ORPER 0 BEFORE BLUNDER REMOVAL 

DU{UiV)b 34,34 
DV«U»V)» 8,991 


U 

V 

DU 

DV 

COR 

DU err 

DV err 

1 1 er r 0 p 

'60 

-60 

30,16 

10,73 

,68 

-4,17 

1,73 

4,52 

>60 

-20 

32,01 

20,73 

.87 

-2 , 33 

11,74 

11,97 

>60 

20 

32,00 

10,00 

.00 

-2 ,34 

1,01 

2.55 

60 

60 

32.88 

9,81 

.74 

-1,46 

,82 

1,67 

20 

—60 

33,92 

4.89 

,25 

", 41 

-4,10 

4,12 

20 

-20 

31,77 

8,70 

,88 

-2,56 

-.29 

2,50 

>20 

20 

31,07 

7,50 

.82 

-3,27 

-1,49 

3,60 

20 

60 

30.87 

6,66 

,87 

-3,47 

-2,33 

4.18 

20 

-60 

37,17 

7,68 

.77 

2,63 

-1,31 

3,12 

20 

-20 

36.13 

8,69 

.79 

1,79 

-,30 

1,81 

20 

20 

35.65 

6,68 

.71 

1.31 

-2,31 

2,66 

20 

60 

35,18 

6,57 

.85 

.85 

-2,42 

2,56 

60 

—60 

3B,00 

7,00 

,00 

3,66 

-1,99 

4,17 

60 

-20 

41,11 

6,00 

,53 

6,77 

-2,99 

7,40 

60 

20 

41,63 

9,23 

,51 

7,29 

,24 

7,29 

60 

60 

39,10 

5,46 

,30 

4,76 

-3,54 

5,93 


TOTAL weighted RMS ERROR* 5,25 
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FIGURE 3-1 RESIDUAL ERRORS FROM TABLE 3-1, 


TABLE 3"2 

POLYNOMIALS FIT TO ORDER 1 BEFORE BLUNDER REMOVAL 


DU(Ui 

V)s 

34,70 + 

,7638®-i*X 

-.8829® 

-2*Y 



DV(U» 

V)3 

8,783 - 

,5617®'1*X 

-,1997® 

-1»Y 



U 

V 

DU 

DV 

COR DU err 

DV err 

1 ! error 1 1 

-60 

-60 

, 30,16 

10,73 

,68 

- , 56 

-2,63 

2.69 

-60 

-20 

32.01 

20,73 

,87 

1,64 

8,18 

8,34 

-60 

20 

32,00 

10,00 

,00 

1..98 

-1,75 

2,64 

-60 

60 

32,88 

9,81 

,74 

3,21 

-1.15 

3,41 

-20 

-60 

33.92 

4.89 

,25 

,14 

-6,21 

6,21 

-20 

-20 

31,77 

8,70 

,88 

-1,66 

-1 , 61 

2.31 

-20 

20 

31,07 

7,50 

.82 

-2,01 

-2 , 01 

2,84 

-20 

60 

30,87 

6,66 

,87 

-1,86 

-2 , 05 

2,76 

20 

-60 

37,17 

7,68 

.77 

,33 

-1 • 18 

1,22 

20 

-20 

36,13 

8,69 

.79 

- , 36 

.63 

.73 

20 

20 

35.65 

6,68 

.71 

-.48 

-.58 

.76 

*20 

60 

35,18 

6,57 

.85 

- , 60 

.11 

.61 

60 

-60 

38,00 

7,00 

.00 

-1,90 

.39 

1.94 

60 

-20 

41.11 

6,00 

,53 

1,56 

,19 

1.58 

60 

20 

41,63 

9,23 

.51 

2.44 

4.21 

4,87 

60 

60 

39,10 

5.46 

.30 

.26 

1,24 

1,27 


TOTAL WEIGHTED RMS ERRORS 3.51 



8 


FIGURE 3-2 RESIDUAL ERRORS FROM TABLE 3-2 


TABLE 3«3 

POLYNOMIALS FIT TO ORDER 2 BEFORE BLUNDER REMOVAL 


OU(Uf V)= 33,35+,085l0»X+,8a58p-3#X2 

-,012B0«Y-,38679«3#X«Y+i6157?-4#Y2 

OV(U» V)= 8,861-,05670*X+,9649(?-3#X2 

",7672(?-2*Y+,2504<?'-3*X<»Y-,9l309-3*Y2 


1 

u 

V 

DU 

DV 

COR 

DU «rr 

DV err 

i terror 

f 

-60 

-60 

30,16 

10,73 

,68 

-,84 

-3,09 

3,20 

r 

••60 

-20 

32,01 

20.73 

.87 

,77 

4,90 

4,96 

■r 

-60 

20 

32.00 

10,00 

.00 

,33 

-4,92 

4,93 

1; 

-60 

60 

32,88 

9,81 

.74 

,58 

-1 • 28 

1,41 

f 

-20 

-60 

33,92 

4,89 

.25 

1,42 

-2,96 

3,28 

y 

-20 

-20 

31,77 

8,70 

,88 

-,35 

-1,57 

1,61 

i-'- 

Y 

-20 

20 

31,07 

7,50 

.82 

, 86 

-2,26 

2,42 

3^' 

-20 

60 

30.87 

6,66 

,87 

-1,07 

,33 

1,12 

w 

20 

-60 

37,17 

7,68 

.77 

,33 

2,69 

2,71 

'1 

20 

-20 

36,13 

8.69 

,79 

,29 

,89 

,94 

■1 

20 

20 

35,65 

6,68 

.71 

,63 

-1,02 

1,19 


20 

60 

35,18 

6,57 

,85 

,77 

1.91 

2,06 

’ ■'! 

60 

"60 

36,00 

7,00 

,00 

-6,01 

1,80 

6,27 

I 

60 

-20 

41,11 

6,00 

,53 

-1.27 

-2,42 

2,73 

• 

60 

20 

41,63 

9,23 

.51 

,67 

.51 

,85 

1 

60 

60 

39,10 

5,46 

,30 

-.62 

- , 63 

,89 

i 

TOTAL 

WEIGHTED RMS 

ERRORS 

2.43 





CU[13« 33.35,CVU]= 8,06 
2, STD Dey= 2,H3 
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FIGURE 3-3 RESIDUAL ERRORS FROM TABLE 3-3 


TABIE 3-4 

POIYNOMUUS FIT TO ORDER 1 AFTER BLUNDER REMOVAL 


DU(U, 

V) = 

34,04 + 

,1013*X - 

,02093#Y 




DV(U» 

V) = 

7.738 - 

,0256»X - 

,01850*Y 




U 

V 

DU 

DV 

COR DU 

err 

DV err 

1 1 9P ror 

-60 

-60 

30,16 

10,73 

,68 

,95 

,34 

1,01 

-60 

-20 

32,01 

20,73 

,00 

3,63 

11,06 

11,66 

-60 

20 

32,00 

10,00 

,00 

4.46 

1,10 

4,59 

-60 

60 

32,88 

9.81 

,00 

6,17 

1,64 

6.39 

-20 

-60 

33,92 

4,89 

,00 

,65 

-4.47 

4,51 

-20 

-20 

31,77 

8,70 

,88 

- , 66 

,08 

,67 

-20 

20 

31,07 

7 , 50 

,82 

-.53 

-.38 

.66 

-20 

60 

30,87 

6,66 

,87 

.11 

- , 48 

,49 

20 

-60 

37,17 

7,68 

,77 

-.16 

-.66 

.68 

20 

-20 

36,13 

8,69 

.79 

-,36 

1,10 

1,15 

20 

20 

35,65 

6,68 

.71 

- , 00 

-.18 

.18 

20 

60 

35,18 

6,57 

.05 

.37 

,46 

.59 

60 

-60 

38,00 

7,00 

,00 

3,38 

- , 31 

3,39 

60 

-20 

41,11 

6,00 

.53 

.56 

-,57 

,80 

60 

20 

41.63 

9,23 

,00 

1,92 

3,40 

3,90 

60 

60 

39,10 

5,46 

.30 

,23 

.36 

.43 


TOTAL WEIGHTED RMS ERRORS ,72 
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CUtU- 3^,Q4*CV[1]= 7,7^ 
n* l/STU DGV* .72 


FIGURE 3*4 residual ERRORS FROM TABLE 3-4 



TABLE 3"5 

polynomials Fit to order z after blunder removal 

DUtUf V)= 33,51+,1015*X+.2499 gi-3*X2 

",0195l*Y+.16B5(F«-5#X*Y+,1330ei~3*Ya 

DVtUf y)= 8,003-,02l22»X-,l949ei-3*X2 

-,01986*Y+,3l080"3*X*Y". 76190-4* Yg 


u 

V 

DU 

DV 

COR 

DU err 

DV err 

1 1 B r ror i | 

60 

-60 

30.16 

10.73 

.68 

,19 

.11 

.22 

60 

-20 

32,01 

20,73 

.00 

3,24 

11,42 

11.87 

60 

20 

32,00 

10,00 

,00 

4.02 

2,23 

4,59 

60 

60 

32.08 

9,81 

.00 

5,25 

3.82 

6,49 

20 

-60 

33.92 

4,89 

,00 

.69 

- 4,75 

4.80 

20 

-20 

31.77 

8,70 

.88 

-.25 

-.14 

.29 

20 

20 

31,07 

7,50 

.82 

-,18 

-,30 

,35 

20 

60 

30,87 

6,66 

,87 

-,02 

,15 

,15 

20 

-60 

37,17 

7,68 

.77 

-,12 

-.37 

.39 

20 

-20 

36,13 

8,69 

.79 

,04 

.95 

,95 

20 

20 

35,65 

6,68 

.71 

,35 

-.52 

.62 

20 

60 

35.18 

6,57 

,85 

,23 

.16 

,29 

60 

— 60 

38,00 

7,00 

,00 

- 4,14 

1.17 

4.31 

60 

-20 

41.11 

6,00 

.53 

.17 

-.02 

,17 

60 

20 

41.63 

9.23 

,00 

1,46 

3,25 

3,57 

60 

60 

39.10 

5,46 

,30 

-.72 

-.23 

.75 


TOTAL WEIGHTED RMS ERROR= ,A,7 



FIGURE 3-5 residual ERRORS FROM TABLE 3-5 
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figure 3"6 residual ERRORS FROM POLYNOMIALS 

FIT TO ORDER 1 AFTER BLUNDER REMOVAL 

DU(UiV)s 34,20 + ,1083*X -,0141*Y 
DV(U,V)5 7,297 - ,82l9acs-2*X -,6235(a-2*Y 



figure 3-7 residual errors FROM POLYNOMIALS 

FIT TO ORDER 2 AFTER BLUNDER REMOVAL 

DU(U, V)= 33,20+,1174*X+.57a9(!>-3*X2 

-,0150*Y+,1627P-4*X*Y+.1373e-3*Y2 

DVtU, V)= 7,294-,8736P-2*X-,8608CJ-4*X? 

-,5937P-2*Y+(6l22(!i-4*X*Y+»5l04(is-4*Y2 


5 ^" 




III.F REN0RHALI2ATI0N OF ONE IMAGE USING THE MISREGISTRATION 
MODEL 

Ones the aeometrlc mlgraglstratlon between two 
norma 1 1 zed Imaaes A and B haa bfsn detected and modeledi a 
new Image can be generated using this modal such that B' 
Is In geometric alignment with Image A according to equation 
3,3, In praotlce» th U reoulras the substitution of 
u+du(u,v) for u and v+dvfuiv) for v In equations (2,12> to 
<2.17) , 

Figures 3»9a to 3»l0b show the results of the 
registration techniques presented In this chapter, Figures 
3»9a and 3”9b are the same as figures 2-6a and 2«6b, Figure 
3*»l^a Is the same as figure 3-9a, Figure 3*»10b shows the 
result of renormalizing 7F78 to align w|th 7F75» using the 
second order registration polynomials from figure 3-7, 





•^iQure 3*9a 
7^75 Unaligned 


flgufe 3»9b 
7F70 Una | I gned 


P" I 9u re 3-10a 
7F75 All gned 


Figure 3-10b 
7F78 A I I gned 






CHAPTER IV 


PHOTOMETRIC N0RMAU2ATI0N BY "DEAD RECKONING” 

Tho photometric norma I I zat Ion problem |s to determine 
two functions RKuiv) and R2(u,v) which define the 
reflectivity of the scene at point tUiV) |n the respective 
Images (and therefore at point T(U|V) |n the acene)t The 
reflectivity function can be determined by "dead reckoning” 
(from accurate calibrations and models) If the response of 
the vidicon and the reflectivity function of the scene as a 
function of the location In the scene and the I I | uml nat I on» 
viewf and phase angles are precisely known, Images can be 
Dhotometr leal |y registered with respect to some type of 
errors In the photometrlo model, As for the geometric 
mode I f the combination of the dead reckoning and 
misregistration models Is used to define the reflectivity 
f unct Ion R{ Ui v> I 

Given the geometric and photometrlo normalization 
functions (RiiTl) and (R2iT2) for a pair of Images, the 
picture comparison problem becomes one of comparing the 



reflectivities of corresponding points in the Itnagesi lieii 
for all point pairs (ul# vl) t Iu2i v2) such that 
T1 ( ulf vl > = T2{ u2| v2 ) , compare RKulivl) with R2£u2rv2), 

IV, A PHOTOMETRIU MODEL FOR THE SCENE 

The scattering of light by a surface Is a very complex 
function of the spectral oomposltlon and polarization of the 
Incident light, the composition of surface materlali and the 
various angles between the Incident light and the surface 
(angle 1) the emergent light and the surface (angle 6) and 
between the incident light and the emergent light (phase 
angle), For some surfaces, w|th known light sources, the 
reflectance function Is fairly well understood. 

The reflectance function of the moon can be modeled by 
the Mlnnaert law (Minnaert C19613)! 

Eout/E|n * cos(t> 5 A * ( cos C t ) *cos ( 1 ) ) fk (4,1) 

where A Is the albedo of the surface, Eln and Eout are the 
light energy incident to and emergent from the surface, and 
k Is a parameter particular to the surface and the phase 
ang I e , 

Young hes found (Young C19713) that regions of Haps 
covered by the Mariner 6 and 7 pictures obey the Mlnnaert 
law fairly we | | J ha found values for k ranging from ,46 to 

6z 






»7lf and va|u«5 for A ranfllng from »07i to |146i 

Poilaok has shown <Pe|laok C1969]) a phase angle 
dependence for the value of ki Although actual data exists 
for only a limited number of different phase anglesf Po||f 4 ck 
suggests that It Is reasonable to Interpolate the following 
table: 

TABLE 4-1 


Bbfisa 

aosie 


0 

deg, 

.5 

50 


,7 

180 


1,0 


IV, B INTENSITY MODEL BETWEEN DATA POINTS 

To compare two Images {F1,T1) and (F2»T2) which were 
taken from from different camera positions and orientations 
It is necessary to compare pixels |n image 1 with those 
pixels In Image 2 which correspond In the scene, For 
discrete (mages, exact pixel co r respondenoe does not exist 
since the Projections of the pixels from the Images onto the 
Planet will not match In location, size and shape, and since 
the orientation of the Image coordinate systems (scan 
direction) may be dlfferenti 

Consequently, fn order to compare such Images it ]s 
necessary to model the Intensity function between given 
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Pixels (Roaanfeld C19693, pp20-21), 

2ER0 ORDER MODELS The simplest model for the Intensity 
function F at non**grld points Is the 0th order modal 
F0(u» v) I 

F0(uiv) s F(hJ) (4,2) 

where (UJ) Is the closest sample point to <u»v), If one 
assumes that F Is defined with (liJ) spaced on Integer 
valuesi then the 0th order medal F0 Is equivalent toJ 

F0(u,v) 5 F(|:u'^.&3fCv+,53) (4,3) 

where Cx3 means the greatest Integer which does not exceed 


FIRST ORDER MODEL* The 1st order model FlCUfV) Is 
bilinear Interpolation which comput'^a a weighted average of 
F at the four sample points which are nearest to (UfV)| 


Fl(uiv) = wl*F(I»j) +w2»F(i+l,J) 

+w3»F(|,J-H) + w4*F( 1+1, j+1) 


where 

| = i:u]» 

p = U"Cu3 , qsv**[:v3 
wl= ( l"p ) + C l~q ) 
w2s(l-p)*q 
w3=p+ ( 1"Q ) 
w4=p*q 


(4,4) 

£ 4 , 5a ) 
£4, 5b) 
£4, 5c) 
C4,5d) 
£4,5 b) 
£4,5f ) 
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Noiji that 0SwCI3Sl and wl^w2+w3+w4«i, 


HIGHER ORDER POLYNOMIAL MOOELSI If hlghar OPdar 
surfaces are used for Int-arpo (atloni Instead of the bilinear 
model# one can require either that aaoh plaoa of surface 
pass exactly through a large number of aample-colnt 
Intensities# or altarnatlvely# one can require that the 
B|®ces agree# not only In Intensity, but also In the values 
of the various derivatives# along their curves of 
Intersection, so that the resulting Interpolated surface Is 
not only continuous but also smooth, 

SAMPLING THEORY MODEL* From a sampling theory point of 
view# a discrete Image can be thought of as a bandwidth 
limited function of two variables# where the point spread 
function W determines the maximum frequency contained In the 
discrete Image, Given this model# the function F can be 
exactly reconstructed from the sample points using a 
2-dlmons!onal generalization of the Shannon sampling theorem 
(Shannon C19493, Prosser C19663 pp, 574-584), However, 
this raathemat I oa I I y sound model for F U -jeed | ng | y 
expensive computational |y# and of questionable practical 
Improvement ever simpler models for the Intended 
app I I cat I on , 

If It were necessary to measure the geometric positions 
of edges of features suoh as craters w|th errors of 
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fractions of a plx«|» than a higher order model would be 
necessary, Fop the Mariner '71 variable features problem* 
photometric accuracy |s more Important than auoh geometric 
accuracy* and consequently* the first order mode] (bilinear 
Interpolation) wss selected because of both Its 
computational simplicity and Its smoothing properties, 




chapter V 


ACCURATE photometric REGISTRATION OF N0RMAU2ED IMAGES 

V,A FORMAL DEFINITION OF PHOTOMETRIC MISREGISTRATION 

Aaaume that wa have two photometrically normalized 

ImaQas wjth raf|ectanca (albedo) functions RKUfV) and 
RE(UfV)i which are also geometrically normalized and 

registered as desorfbod In Chanters II and UI, Assume also 
that the asSuai scene ref|eot|v|ty la the same top the two 
Images* The . Images are said to be "photomstr I ca ! i y 
mlspeglstered relatively” at point Cu»v) |f: 

RKuiV) ?! R2{u,v) (5,1) 

If R(uiv) Is the true scene ref|ect|vjty then an Image R1 Is 
said to be "photometrically m | speg 1 stored absolutely" at 
point (uiv) If! 

Rl(u»v) ?! R(u*v) (5,2) 

The following sections present a mqdel for photometric 
misregistration and derive functions which describe this 
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mlsrealstpatlon, 


V.B HQDEL FOR MISREGISTRATION 

To Photometrically registop Images more accurately than 
Is possible with the »dead reckoning” photometric 
norma I I zat I on requires that we have a model for photometric 
misregistpatloni There are many potential sources of 
Photometric errors In Images which have been “normalized” by 
Chapter IV| Some of these arei 

1) Camera errors 

a) Errors In light transfer function 

b) Non~Lin|form sensitivity across vidloon surface 
e) Residual images 

d) Coherent noise 

e) Random noise 

2) Photometric model errors 

a) Reflectance function variations with position on planet 

b) Inappropriate values of k In equation (4,1) 

A general model for photometric misregistration would 
be to assume that there |s a function P which describes for 
each picture the combined photometric characteristics of the 
surface of the pianetf the response of the v|d!con» and the 
photometric model of Chapter IV, If the photometric errors 
are spatially uniform? that |s, the errors do not depend on 
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theiP loaatlon on tho aurfaoa of either the vidicon or the 
Dianeti then the photometric registration function P Is a 
function only of the pefleotivlty R(UfV) of each point, In 
general, P Is a function of a set S of variables, sueh as 
the position on the vldloon surface, the position on the 
Planet, and the Illumination, view, and phase ancles as well 
as Rtu,v), Then, for two photometrically normalized Images 
and R2, we would have two Photometric registration 
functions PI and P2 such that! 


PltRl(u,v)iS)) s R(u»v) s P2CR2(u,v),S) 


(5,3) 


where R(u,v) |s the true reflaotivlty (albedo) of point 
(u,v), If It Is not necessary to obtain photometric 
registration to true reflectivities, then the Images can be 
photomotr 1 cal |y registered relative to one anotnep using the 
function P12 which is defined* 


P12(r»S> = Pi' (P2( P,S),S) 
where PI' Is the Inverse of PI, or 


(5,4) 


Ri(u,v) = P12(R2(u, V) ,S> 


(5,5) 


The Mariner 6 and 7 pictures have been processed 
(Rlndflalsch C19713) by the Jet Propulsion Laboratory (JPL) 
to remove as many of these errors a® possible, The h|gh 
level of coherent noise present |n the Mariner 6 and 7 
pictures required that JPL develop some rather sophisticated 
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selective flltei'lng techniques, There |s soma question 
about the absolute photometric accuraoy of these final 
pictures, and thepe are still errors due to the residual 
Image processing in the area of the I |mb of the planet, 

There does not presently exist an aoeurate mode! for 
the photometric errors present In the Mariner '69 pictures, 
However, after comparing Images which were procossad by the 
techniques of Chapters II, III, and IV, It was found that 
the predominant errors were due to errors in absolute camera 
sensitivity, Consequently, the following simple photometric 
registration function was used* 

Pl2<r,s> = a*r + b (5,6) 

where the parameter a adjusts for errors In camera 
sensitivity, and parameter b adjusts for any constant 
offset. This model assumes that a and b are constant for 
each point in Images R1 and R2, 

It might be better to allow a and b to be polynomials 
In the two variables u and v. Such a mode) would correct 
for errors which are spatially dependent, If It is possible 
to empirically derive a(u,v) and □<u,v) and |f we could 
separate the contributions due to the camera from those due 
to the photometric mode) of the planet, then the empirical 
functions should be useful In refining our photometric modal 
for Mars, 
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v,c A measure of photometric misregistration 


Obvlouslvf fn any discussion of arpors betwaan sets 
poln'is It Is necessary to define some measure of the "tota, 
orror"t The conventional RMS error measure was chosen 
primarily because the familiar least squares technloues 
fflinlirize that measure, 

The RMS error In photometric registration between two 
pictures Rl and R2 Is then defined! 

RMS(R1»R2) = sqrt { sum URKujv) •> R2(u»v))t2) / N) (5,7) 

u» V 

where N Is the number of points )n the picturesi 

V,D choosing a model to minimize photometric MISREGISTRATION 

The purpoBo of this section Is to empirically derive 
the function P12 which '‘best'* registers pictures Rl and R2 
photometrically In terms of the RMS error. More precisely, 
we find P12(r#S) such that! 

RHS<R1,P12(R2,S) ) = min (5,8) 

or 

sum ((RKufVj - P12(R2(uiV),S))»2) = m|n (5,9) 

U» V 

for the simple model of equation (5.6), minimizing 
equation (5,9) Is equivalent to minimizing! 
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E(Rlt2J + at2*E(R2^2) - 2a»E(Rl«R2) - 2b*E(Rl) 

+ 2ab*HR2) b^2 s m(n C5.10) 

where E Is the expected valutt The solutions for a and b 
are : 

a = cor(Rl,R2) * sd (RlJ/sd CR23 (5,11a) 

b = E<RD ~ a*E(R2) (5,Ub) 

where ad Is the standard davlatlon# and cor Is the cross 
correlation defined by equation C3|5), 

If a more complicated model |s used, where a and b are 
functions of the (u,v) position |n the Image formed by 
linear combinations of functions Ak(U|V) and Bk(u,v), then 
we want to find coefficients aCk] ana bCk] to minimize* 

sutr (CRKufV) • R2(u,v) * sumt aCk J*Ak ( u* v) ) 
u, V k 

- sum(bi:k3»Bk(u, V) ))t2 k min (5.12) 
k 

where Ak and Bk are Indexed by k. The values of the aCk] 
and bCk] are found In the traditional manner of [east 
souares, Intuitively, first order polynomials In u and v 
appear to be useful for the functions A and B, Since this 
technique for photometric registration has not yet been 
US 0 O, the proper order and form for the A and B functions Is 
not yet known, Only zero order models (equation (5,6)) have 
actua I I y bean tried, 


CHAPTER VI 


differencing and difference analysis 

The tttl6 of this dissertation Is ’'Computer Comparison 
of Pictures"! but the previous four ohapters have all dealt 
with Imaoe normalization and rsg I stpat I oni and nothing has 
really been said about Image comparison* Of coursoi a|| of 
these preliminary operations were neoessary so that the 
differences between the t..j pictures represent actual 
changes In the scenes rathep than differences d' 3 to the 
eonoltlons of viewing and 1 1 I um| nat t on* 

This chapter describes teohniaues for Image 
differencing and the analysis of these differences, 

VI, A POINT BY POINT DIFFERENCES 

The most obvious form of differencing of two normalized 
and registered pictures R1 and R2 Is the point by point 
difference defined byS 

D(u»v> = RKuiV) - R2Cuiv) (6,ia) 
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If the two Images have net been ohotometr I ca I | y registered/ 
then the following point by point difference also performs 
the photometric registration! 

U(u»v) = RKuivJ " A(u»v)*RH(ui v) • B(u»v) t6ilb) 

where A and B are the photometric misregistration 
polynomials derived In Chapter V* These difference pictures 
show where there are errors In the data dye to noise of 
various forms/ where there are errors In the phntometrlo or 
geometric registration mode|s» op» hopefully/ where there 
are actual differences In the reflectivity of the scene, 

Applying the latter differencing technique to the 
pictures In figures 6-la and 6-lb (whjch are the same as 
3-10a and 3-10b)/ we get the difference pictures which are 
shown In figures 6-2a and 6-2bt Figure 6»2a was produced by 
subtracting figure 6-lb from 6-la (and adding a constant so 
that there are np negative numbers), Figure 6-2b Is the 
negative of 6-2a, 

The most obvious feature of these difference pictures 
Is the large Irregular bloD In the lower-right oorno'-/ which 
is dark In 6-2a and bright In 6»2bi and whose cause will be 
explained |ater. There are numerous small circular light 
and dark spots which are due to reseau marks on the 
vidlcons* (In future work/ these areas will always be 
black), Reseau marks are deposits on the v)dlcon surface 
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^iQure 6-la 
7F75 Aligned 


P"igure 6“lb 
7F78 A I I gned 


Figure 6'2b 
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Figure 6-3a 
7F75 Aligned 


Figure 6-3b 
7F77 A I Igned 


Figure 6-4^ 
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for geoiretrlc alignment of the vidioon scan, Thepe are also 
light and dark linear horizontal stpeaKs which are oriented 
In the horizontal scan direction In the original camera 
coordinate system, These stpeaks are probably due to 
coherent noise In the camera system CR|ndf|e|soh C19713J, 
Slight geometric misregistration Is Indicated around N|x 
0|yirp|ca, the large crater located at the top-oenter of the 
aligned pictures. A high |eve| of visually uncorrejated 
noise Is also present. The global dark to light shading 
frorr loft to right In figure 6-2a can be explained by errors 
In the assumed dependence of the photometric function on 
viewing angle near the limb, A linear model for the 
photometric registration model of equation (5,12) would have 
reduced this error, 

The only significant unexplained difference between 
figures 6-la and 6-lb is the large blob which is brighter In 
6-ib than In 6-ia, This blob has been attributed (Leovy 
C19713) to a cloud consisting of water vapor, This 
dlf^'"' a Is visible In figures 6-la and 6-lb, but Is very 
difficult to see in the original pictures (figures 2-5a and 
2«5b) , 

Figures 6-3a through 6-6b show the results of 
introducing a third picture named 7F77, which was the 
picture taken Immediately before 7F78 In the Mariner 7 far 
encounter sequence, Picture 7F77 was normalized to the same 
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orthographic projection ae was used for 7F7& and 7F70> and 
geoiretr I ea I I y aligned with 7F75, The figures show 
dlfferenoes between 7F77 and both 7F75 and 7F78| Since 7F77 
was taken between 7F75 and 7F70 |n tlme» It la reasonable 
that we see an Intermediate state of the development of the 
0 I oud I 

The higher level of noise In figures 6-6a and 6-6b Is 
due to an Increased gain factor Introduoed by a histogram 
•'stretch 1 ng” program, This stretching Is used to adjust the 
Intensity range of a picture to maximize the contrast whan 
generating a photograph, These figures also show another 
camera related error across the top which Is oriented In the 
horizontal scan direction of the camera, It should be 
emphasized that these pictures were never geomotr I oa | I y 
registered with one another, but both were registered to the 
3009 third Picture (7p75), There seams to be tittle op no 
extra geometric misregistration due to cumulative errors as 
might be expected, 

VI, B NOISt "REMOVAL" 

The high levels of noise present |n the difference 
pictures suggests that some form of noise removal Is needed. 
The obvious problem with noise removal Is that usually one 
cannot distinguish signal from noise, and oonsequent t y, when 
performing noise removal one also oepforms soma s|gnaj 
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If the noise !s random, such that the noise In adjacent 
samples Is unco r re I ated t then one way to reduce It Is by 
spatial filtering or averaging! Figures 6-7bi 6-8b, and 
6-9b show the results of replacing each point In the 
difference pictures (6-7ai 6-8a, and 6-9a3 by the local 
avarags computed over an llxli**poInt window. This process 
effectively reduces the level of random noise, but the 
cloud, and the systematic errors In photometry still remain, 

Other types of no|se removal have been developed 
elsewhere but have not yet been tried here, One such 
technique Is ''salt and pepper” removal, which consists of 
replacing points which differ greatly from a surrounding set 
of approximately equivalent points. This technique (which 
is soiretimes called "Custer |ng” after the well known 
American general, who, when surrounded by Indians, was wiped 
outJ would remove most of the random "shot” noise In the 
difference pictures without Introducing the blurring created 
by spatial averaging, 

VI, C contouring iSO-niFFERENCE UEVEUS 

The technique of oontcviring Is often used In 
terrestrial map-making to ^Mphaslze lines of constant 
altitude, Recognizing the utility of this technique. 







r I gure 6-7d 
7F78 - 7F75 Blobs 


Figure 6-7c 
- 7F75 Contour ed 












figures 6-7c» 6-8ci and 6«9o wepe produced by contouring the 
averaged pictures 6«7b, 6«8bf and 6«9b Ci3» 

In these fIgureSf the sharp dapk to white edges ociur 
at difference levels of 16; 3^1 and 4B In the corresponding 
spatially averaged plctursa whose difference values range 
front 0 to 63, Comouta'c 1 ona 1 i y » these pictures ware 
nriwlally generated by multiplying each difference sample by 
•> , and trj'ieatlng the result to 6 bits (0 to 63 range^i 

These iontegr pictures fairly well outline the cioud> 
but they also shew a lot of Unas In areas which seem to 
contain no significant differences, In fact, one of the 
contour levels corresponds to the mean value of the 
dlfforepce picture, and therefore, one would expect many 
oont'^urs corresponding to small positive and negative 
deviations from the meai'i 

An Improved contouring algorithm Is planned which will 
select contou.'* levels on the basis of the histogram of the 
difference picture, attempting to emphasize significant 
differences, but not small deviations from the mean, 


C13 The spatially averaged pictures were used here because 
of the high random noise levels }n the unavepaged pictures, 
which, when contoured, resulted primarily In garbage, 


VI, D DIFFERENCE «BLOB« DETECTION 


Another 

form 

of analysis of 

d 1 f f erence 

p I ctures 

I s 

d I f f arenoe 

blob 

detection, which 

attempts 

to find 

the 

out 1 (naa of 

areas 

(blobs) of significant differences, 

The 


stgnificanc* of a b|ob Is dotermlned by the magnitude of the 
diffarsnoao, and by the length of the edao surrounding the 
difference blob. 

Figures 6**7d, 6«8d» and 6-9d show the results of 
applying suoh a blob deteotor to the corresponding averaged 
pictures 6-7b, 6-8b, and 6»9b, 

A further refinement of the blob detector (Hannah 
C1971J) attempts to determine which of the und I f f erenced 
pictures contains the difference blob. This determination 
Is made by looking |n the und ( f f epenced pictures for edges 
In the vicinity of the edges of the difference b|ob, An 
edge Is defined statistically In both pictures |n terms of 
the means and variances both Inside and outside of the 
difference blob, If the variance of one picture ]s 
significantly larger than that of the other (In the vicinity 
of the b!eb)» then the blob Is attributed to the picture 
with the larger variance, Otherwise/ |f the absolute 
difference In the means Inside and outside of the blob Is 
larger In one picture than In the other, then the blob Is 
attributed to the picture with the larger difference. This 
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technique Is still be|ng deve|opedi and will be valuable to 
dlffepenee analysis, 


Other difference analysis techniques ape plan 
not yet Implemented, 


ned» but 
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CHAPTER VII 

COMPUTER IMPLEMENTATION 

The techniques described In this dissertation were 
developed for use on a digital computer, Analog 
Imp i ementat I on of these techniques Is very limited because 
of the difficulty In performing the necessary photometric 
and geometric normalizations and registrations using analog 
dev Ices, 

Except where otherwise notedr all of these techniques 
have been Implemented by the author using the PDP-10 
computer at the Stanford Artificial Intelligence Project, 
This computer Is attached to 128K (IK s 1024) 36«blt words 
of core memoryi and executes Instructions at the rate of 
approximately one every 3 microseconds, The POP-10 Is 
"time-shared" among typically 20 users at a time, making It 
Dossible for many users to develoo and debug programs 
Interactively wjth very short "turn-around" times, 
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Most of the programs have been written In SAIL 
(Swjnehart [1970]), a local d|a|ect of ALGOL-60, and many of 
the time consuming Inner loops have been written |n the 
assembly language embedded In SAIL, 

VII, A BLOCK description OF THE PROGRAMS 

The programs are divided both logically and physically 
Into blocks which operate Independently, Most of the major 
blocks of figure 7-1 correspond to the techniques described 
In Chapters II through VI, 

Images are represented as packed arrays of light {or 
albedo) values, and are accofssed using PDP-10 byte 
operators, As many samples as possible are packed |n a 
word. Usually, ejther 6 or 9 bjt samples are used, giving 
either 6 or 4 samples per 36-blt word, Thus, an Image 
consisting of 200 samples per line, 200 Ijnes, and 6 samples 
per word would require 6667 words of storage, Procedures 
are available to transfer pictures between core and disk, 
and to display Images on either of two video synthesizers 
(section V n • B ) , 

In the block diagrams, rectangular blocks Indicate 
functional operations, and oval b|ocKs Indicate data, The 
description of each block follows! 
















VII. A, 1 NORMALIZE 


normalize !s the name of the b|ocH which Implements the 
norrra M zat I on techniques of Chapters II and IV, The 
Internal structure of NORMALIZE Is shown |n some detail In 
figure 7-2, The major functions of NORMALIZE are geometric 
and photometric normalization based on models for camera and 
scene geometry and photometry, 

USER interaction: The block labelled "user Interaction" 


Indicates control of 

the f 0 1 lowing 

Parameters 

for 

the 

orthographic projection! 






1) 

Direction (center 
project 1 on, 

latitude and 

longl tude) 

of 

the 

2) 

Pixel size In the 
pixel), 

projected Image 

C k I 1 ometers 

per 

3) 

Horizontal and vertical size 
projected Image, 

( 1 n 

pixels) 

of 

the 


4) Rotation of the projected Image, 

5) Position of the center of the projected image on the 

0 i anet , 

The combination of the models and the user Interaction 
generates a set of projection oarameters which control the 
actual normalization procedure labelled "PROJECTOR", 

projection ANALYEEr: This block analyzes the geometric 

calibration data for the spacecraft Cl.e. spacecraft 
position and orientation at the t|me the picture was taken) 
to determine the area on the surface whioh Is covered by the 
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Figure 7-2 Block Diagram of NORMALlaiE 
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Dioture, This analysis is usually done for pairs of 
pictures to determine the surface area whioh |s Included In 
both clotures, A graphical display of the spacecraft and 
Planet geometry Is presented to tne interactive user to 
enable him to specify the desired projection, 

AVGWINI Since the Mariner '69 pictures are quite large <180K 
words} It Is necessary to operate on windows of entire 
nlcturesi spatial averages of Pictures# or combinations of 
windows and averages, AVGWJN computes an Nx by Ny spat|a| 
average of an arbitrary rectangular window of a picture, 
where Nx and Ny are the numbers of Pixels averages In the 
vertical and horizontal directions. If pixels of the 
required projected Image correspond In s|ze to a vary targe 
number of pixels In the raw Image, then spatial averaging Is 
mathematically necessary because of the sampling theorem, 
and coirputat I ona I I y useful to reduce the memory size 
required for the raw Image, 

PROJECTORj This block generates a geometrically and 
photometrically normalized picture from a raw or averaged 
raw picture (which Is hopefully correct In light values and 
camera geometry) using the normalization parameters and 
reflectance parameters, For each point In the projected 
Image, PROJECTOR calculates Its position In the raw Image, 
Since this position Is usually not exactly at a sample 
point, a bilinear Interpolation of the jlght values of the 
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four closest points Is ealoulatedt PROJECTOR also 
oalouletes the sum viewf and phase anglesi which together 
with the light value; yield the ajbedo using equation (4,1), 

REGISTRATION POLYNOMIALS! NORMALIZE performs the 
renormalization procedure described In section HliF using a 
pair of polynomials produced by the geometric 
misregistration mode I , 

VII, A, 2 MATCH 

This block Implements the techniques for determining 
geometric misregistration described In Chapter III, The 
Internal structure of this block }n shown In more detail |n 
figure 7«3, Selected nxn windows of two normalized Images 
are cross correlated according to equation (3,5), A search 
Is made for the vector (duU3»dvCI3) which maximizes the 
value of the correlation function for Integer values of du 
and dv, The correlation surface |s modeled by polynomials 
In two variables In the vicinity of the maximum using a 2»D 
least squares program, The maximum value of this surface Is 
found by computing partial derivatives finding the zero of 
the Euclidean norm of the partial derivatives of the 
surface, using a two-dimensional generalization of Newton's 
Method (section III,D), 
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Figure 7-3 


Block Description of MATCH 


vn.A,3 POLYNOMIAL FITTER 


This block Performs a least square fit of two 
polynomials dutUivJ and dv(u»v) to the empirical oorreiatlon 
vectors (duU]fdvcl]) at sample points (ut|]fVU])i The 
values of the oorreiatlon peaks cCl3 are used as weighting 
Qoefflolents to the data points, 

VII, A, 4 PHOTOMETRIC REGISTRATION 

This block ImPiaments the photometric registration 
techniques described In Chapter Vt It computes the 
registration parameters which minimize the photometric 
m I srsa I strat I on between the geometrically aligned Images, 

VII, A, 5 DIFFERENCER 

The DIFFERENCER Implements the p I xe | -by-p I xe I 
diffareneing described In seotjon VI, A, The differencer 
uses the photometric registration model generated by the 
techniques of Chapter V to compute the difference picture 
defined by equation <6,lbl, 

VII, A, 6 DIFFERENCE ANALYSIS 

Th|e block performs the difference analysts techniques 
described In Chapter VI, and Is shown In more detail (n 
figure 7-4, 
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VII, B IMAGE DISPLAY 


Images which are generated by any of the Image 

generating techniques of this dissertation can be displayed 
by either of two Image dlsolay devices, Both of these 

hardware devices are eontrolled by programs which Input 
pictures from the disk file systomi and output the pictures 
on a selected area of the Image generation surface^ which 1n 
each case consists of a cathode ray tube, 

VII, B,1 HIGH RESOLUTION VIDEO SYNTHESUER 

The high resolution video synthesizer consists of a 
collection of digital and analog circuitry to control the 
position and Intensity of the beam of a Tektronix model 611 
storage osq I I I oscope which Is used In r.cn»storage mode, 
This synthesizer generates grey level Pictures one line at a 
time as foil ows : 

A, The 611 oscilloscope beam Is positioned at the (Xfy) 
coordinates of first point on the line, 

8, The 611 oscilloscope beam Is turned on for a duration 
Proportional to the Intensity value for this point, 

C, The beam Is deflected to the right one sample position, 

and control Is returned to step 8 If there are st||i 
points to display on th | s line, 

D, Uhep the line Is finished control returns to step A, 
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The hardware for the high resolution video synthesizer 
Is theoretically capable of 4096- by 4096-po|nt spatial 
resolution with 512 levels of grey, The system Is currently 
limited by the capabilities of the 611 osc I I I oseopof which 
limits the effective resolution to aoproK 1 mate I y 700 by 700 
points, with approximately 64 resolvable levels of grey, 

The amount of time required to display a picture 
depends on both the size of the picture and Its content, 
Bright points require that the beam be turned on longer than 
dark points, Typically, a picture requires 25 microseconds 
per point, This means that a 200 by 200 picture would be 
generated In one second, In order to view pictures which 
are generated so slow|y, a photograph must be taken, a 
P olaroid camera Is used for this purpose, 

VII, B, 2 REAL TIME VIDEO SYNTHESI2ER 

The rea|-t|me video synthesizer consists of a different 
collection of digital and analog circuitry which generates a 
television picture, using as Input B channels from a 10 MHz 
digital disk (Data Disk), These 8 digital channels contain 
the bits of the binary coded sample values, which, when used 
sS Inputs to a digital to analog convertor, generate one of 
256 analog voltages. 

The picture consists of 4B0 scan lines of 512 points 
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per llne» with 2S6 levels of gpey (apopox Imate |y 128 usable 
levels), This picture can be althep viewed directly op 
photographed, 

Vn,C EXECUTION TIMES 

This section describes the CPU time pequlr^id for some 
of the Important and t I me*consum I ng parts of the system (CPU 
time refers to actual processor tlmsf rather than elapsed 
time), These times ape averages ov*r several passes through 
the sybtemf and are only approximate. 


The 

CPU times 

to generate 

the 

200x200 pixel 

p I ctures 

shown I n 

this dissertation are 

as f 0 1 lows 1 


QCQgCSQ} 


Slise 

cooosots 


AVGWIN 

10 

sec , 




PROJECTOR 

60 

sec , 




MATCH 

45 

sec. 

for 

16 oorrelatlon 

wl ndowa 


220 

sec k 

fop 

81 correlation 

windows 

POLYFIT 

4 

sec 

for 

1st order 



There are the on|y programs whose execution times have 
been measured, Nqns of the other Programs require times 
comparable to either PROJECTOR or MATCH, 


chapter VIII 

CONCLUSION 

This final Chaoter describes what has been accomplished 
towards solving the Mariner '71 Variable Features Problem* 
how this contributes to the general science of Image 
processing* and plans for future refinements and 

applications of these technlaues* 

VII I .A accomplishhents 

The primary goal of this research was the development 
of image processing techniques to compare pictures of Mars 
which were taken from space under different conditions of 
viewing and Illumination* The solution was to normalize the 
pictures to the same projection using geometric and 

photometric models for the camera and the scene* Geometric 
and photometric registration techniques were developed to 
remove the effects of errors |n these models. 


Af^ar Pictures have been normalized and registepedi 
they ape analyzed fop dlffepancesi Most of the differences 
which have been found between the Mariner 6 and 7 pictures 
are due to noise |n the eameraSt Some noise reduction and 
difference analysis techniques have been developed to aid 
the analysis of the difference pictures. 

The examples presented In this paoer show the detection 
of a real difference# dge to a o|oud on Mars# The system 
has been tested with other Mariner 6 and 7 Pictures# which 
have not displayed such dramatic and recognizable 
d 1 f f § renoes, 

The Jet Propulsion Laboratory devised a test of the 
capabilities of people and computers to detect variable 
features# JPL Introduced artificial differences of 5Ji In 
reflectance Into a Mariner 6 picture# This modified picture 
was then geometrically distorted to what JPL olalmed was a 
different perspective view of Haps, The test was to sea If 
people tor computers) could determine the differences 
between the original picture and the modified# distorted 
picture, 

The result was that people were unable to see the 
differences, but the system described In this dissertation 
was able find most of the differences, The only differences 
that were difficult to detect occurad because the jpl 
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distortod picture was not a true parspeotlve view of the 
planet# and# after geometric normalization and rea I strat | on# 
some high order resfdual distortions remained, 

No systematic difference analysis of the Mariner 6 and 
7 pictures has been attempted yet, However# this analysis 
Is planned# to test the Mars global model which Is described 
In section VI 1 1 »C , 

VIU.B CONTRIBUTIONS 

The major contributions of this research to the science 
of picture processing are summarized as follows! 

1) Accurate geometric registration based on cross 

correlation! Although cross correlation has been used 
In character recognition and scene congruence 

(Fischler [1971:)# apparently no one has derived a 
model for the misregistration between two pictures and 
used this model to produce two pictures which are In 
geometric alignment, 

2) Photometric normalization using the M|nnaert scattering 

mode I , 

3) Development of a system which combines a large variety of 

Iirage processing techniques! Some of the technlouas 
described In this paper have been used elsewhere, but 
have never baen combined In a coherent system which 


actually works with real ^rather than synthetic) 
D 1 otures I 

4) The development of an Image processlno system for 
Interaction with a planetary exploration mission: 
Although the system has not actually been tested In 
Its Intended appl|cat|on» It should satisfy most of 
the needs of variable features detection, The ability 
to rapidly detect changes on Mars provides the ability 
to Increase the scientific return from the mission, 

VIII. C FUTURE PLANS 

Further research Is planned to refine the techniques In 
this paper* to apply these toohnloues to new Image 
processing problems, and to develop new capabilities for the 
Mariner '71 Mission, 

REFINEMENTS! 

Several Improvements In the correlation maximization 
search technique are planned which wilt use Information 
about the s | gna | - to»no I se ratios and autocorrelation 
functions of the pictures tu adjust various search 

parameters, 

The higher order photometric misregistration model of 
equation 15,12) w|l| be Implemented, This model should help 
to refine the parameter tk |n equation (4,1)) of the 
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Minnaert function, 


Sotre new picture difference operators are planned which 
would rrlnlmlze the effects of one pixel misregistrations, 
This techniaue finds the difference between a point In 
picture A and a point |n picture ti which Is closest In 
Intensity value and a neighbor, An attempt w||| be made to 
Introduce symmetry Into this operator. 

Tests of "significance” of differences need to be 
developed, Presumably, these tests would be based on a 
noise model for the camera, and some knowledge of the types 
of olfferences to expect, 

!i 

j, 

OTHER APPLICATIONS! 

t 

I Research |s planned which would use the cross 

1 

|i correlation registration tachnlgue to match areas of two 

fs 

li parallax views of a scene, Using a set of match points, a 

P 

|j deoth model for the scene will be derived, 

r( 

[i 

I NEW CAPABILITIES FOR THE MARINER 1V71 MISSIONi 

i 

I Additional capabilities are reaulred for the Mariner 

f '71 Mission, To detect variations on Mars which occur over 

i 

f, long periods of time, It Is necessary to maintain a 

I 

I "library" of previous pictures, New pictures would be 

1 

t oomparsd with previous pictures of the same area on the 


surface, 


E.aeh picture contains approximately 5 million bite of 
data* which can bs oomprassed by 3il using Huffman coded 
pixel differences, At least 1000 Pictures with controlled 
eun and view angles are to be processed for differences, 
With the data oompresslonf this amounts to 1,7 billion bits, 
which Is about three times the storage capacity of our 
present disk file system. This quantity of data obviously 
will require a careful |y designed data management and 
retrieval system, 

A data structure Is planned which will be organized by 
area on the surface and by original picture. 

The surface of Mars will be divided Into convenient 
areas, probably 10 degrees latitude by 10 degrees longitude. 
For each area a list will be maintained (on the disk file) 
of the following Information* 

1) A list of ell pictures which cover this area, 

2) A reference Image for this area, This Image represents 

this area to the best of our knowledge. The reference 
Image Is probably created by averaging pictures which 
do not demonstrate significant differences In this 
area, An Image which e^hows the Pixel by pixel 
variances of these pictures should probably be 
associated with the reference Image, 

3) Control points, Control points are features, such as 
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oratersi whose positions are accurately known, These 
points are useful for registering Images to fixed 
surface coordinates, For each control point, the data 
structure should Include Its planet coordinates, and a 
sira I I Image of the control oolnt for use by the 
correlation Procedure, 

4) A list of any significant differences seen In this area, 
For each such difference there might be a difference 
Image, a contour map, a b|ob outline, or some other 
representation of the difference. The actual 

difference Information need not pe currently stored on 
the disk file system, but Information must be present 
to retrieve It from magnetic tape, 

For each picture the following Mat wli| be maintained* 


1) Picture Identification (e.g, 7F75), 

2) Calibration data, I.e, spacecraft position anc 

orientation, and sun angle, 

3) List of surface areas covered by this picture, 

4) Location In the magnetic taoe library of* 

a) Original version of the picture, 

b) Locally processed version of the original, 

c) A list of all other Images which were derived fron 
this picture, and the necessary retrieval data. 


vin.o philosophical remarks 


Som« svntral remarKa about tht Imag# oroceaalng 
philosophy ara an aopropriata conclusion to this 
dissartatlon. There are many difficult proplams In this 
flaldi which can be made either harder or easier depending 
on the approach taken to solve them, Mhl|e It Is recognized 
that models for the camera and scene ape fmportanti one 
cannot depend totally upon accurate calibration of these 
models to eliminate the registration problems# whether the 
working environment Is the surface of Mars» the surface of a 
flat table# or a hlghwayi 

It Is Important that techniques be developed which are 
sufficiently general that they can be applied to 
unanticipated environments, It Is believed that most of the 
techniques In this Paoer pan satisfy this requirement by 
suitably changing the various models# 

The unmanned space exploration programs should provide 
excellent application areas for computer Image processing# 
since# as the spacecraft probe deeper Into space# the needs 
for onboard Image processing will Increase, 

All exploration of the surface of other planets w[th!n 
the foreseeable future must be done by unmanned landing 
vehicles# Any intelligent exploration of other planets will 
reaulre some level of onboard Image processing oapablllty In 
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ord«r that the vehlele can navigate and Interact with its 
•nv f ronirent, 

It Is hoped that the results of this and other research 
In the application of computers will be used by mar to 
better understand the universe around h|mi and to free man 
from the trivia of dally llftp rather than to provide men 
the tool to enslave others, 
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APPENDIX A, 

OEPIVATION OF MATRIX R 


This appendix gives the derivation of the matrix R of 
eauetlon (2.4)« given calibration data which describe the 
position and orientation of the camera, 

The rotation matrix R Is derived In several steps from 
the geometric calibration data, The first step derives a 
rotation matrix R0 whleh rotates the prlnolpal ray Pp of the 
camerai given In scene ooordinatasf Into point C0f0iw) which 
Is assumed to be the center of the image plane, given In 
camera coordinates, 

(0»0, 1 iPpI I = RUPo) U.l) 

This rotation matrix Is easily oonstruoted by first rotating 
Pp In the x-y plane by matrix R1 Into <x',0,2)', and then 
rotating (x<*,0iZ)' |n the x-z Plane by matrix R2 Into 
(0,0fZ')S The matrices R0i Rl» and R2 are defined* 

x/k y/k 0 

Rl = «y/k x/k 0 |A,2> 
0 0 3 . 
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whore k®aqrt( x»2*y*2 ) , 


z/h 0 «k/h 

R2 s 0 1 0 (A, 3) 

k/h 0 z/h 

where h = sqr t( k»2+z »2 ) « sqrt{x«’2+yt2+z»2) . 

Then wa define R0 » R1#R2, R0 Is not unlquoi sinae a 

rotation around the principal ray Is st|!l possible, This 
ambiflulty Is solved by specifying the direction of one other 
vector In both scene and Image coordinates, Given vector V 
In the scene and ths angle « between the u-axis and the 
projeotlon of M on the Image plane the rotation matrix R3 Is 
def I ned I 

R0(V) s (XfyiZ)'i (A, 4) 

If the direction of v Ip the Image plane |s P, then costP) = 
x/|)x»y||i and s!n(P5 “ y/l|x»y|(, To orient the vector 

R0(V) In the direction It Is sufficient to rotate R3(V) 
by an angle of «»|3, Rotation matrix Rd Is then defined! 

cos(«*P> sln<«-P) 0 

R3 = -slnt®"f3) cosCa-p) 0 CA,5) 

0 0 1 

Finally# the matrices R4 and R are defined! 

R4 S R3 # R0 (A, 6) 

R = tnv(R4) < A,7) 







For Mariner 6 and 7 pictures of Mars# the camera position Is 



derived fpom the latitude and longitude of the 
eub-spacecraf t point on the surfeoe^ and the altitude of the 
spaceoraft* The orientation of the camera Is determined 
from the latitude and longitude of point Pp> the 
Intersection of the ppjnolpal optic ray with the surface, 
and the angle In the Image plane between the u«*axls and a 
nor th"Po I nt I ng tangent vector at point Pp projected on the 
Image Plane, If Pp Is given In spherical coordinates ast 

Pp = r * I C08< long)*cosUat) (A, 8) 

iS|n( Iong)*eo8( {at} 

, s jn( lat) ) ' 

where lat Is latitude, long Is longitude, and r Is the 
radius of the sphere (Mars), then the north-pointing tangent 
vector to the sphere at this point |si 

V - < -eos( |ong>*8|n( lat) (A, 9) 

,-s|n( jong)*s|n( lat) 

,cos( lat) ) ' 



APPENDIX B 

THE EPPECTS OF NON-TRANSLATIONAL 
GEOMETRIC distortions ON CORRELATION 

This appsndlx analyses the effects of non-tpans I at | ona 1 
geotr.etrle distortions between the windows of two Imases on 
the value of the correlation function, An upper limit on 
the size of the correlation window la derived as a function 
of the iragnitgdes of the higher order errors and the spatial 
frequency Information In the windows, It Is shown that the 
effects of non-trans latlona I geometric distortion will be 
less If the Images contain primarily lew spatial freauenojes 
rather than high sPat|a| frequenelesi 

A,1 ROTATION AND SCALE CHANGE 

Rotation and Scale factor errors between correlation 
windows can be modeled with a first order misalignment 
mode I * 

y' c a*u + b»v + c * Su * <u*cos(8) + v*sln(«)) 

v' » d«u + e*v + f = &v * {v*co8tP) - u*sIn(P)) {B,lb) 
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whe r« 


o s aretan(b/a) Ctj,3a) 
0 » arctan<d/o) Ui.2b) 
Su = sqrt(at2+bT2) (B,2o) 
Sv 5 sart<dt2*ot2) <B,2d) 


Presumably! the rotation angles e and I3 are small and the 
scale ^actors Su and Sv are approximately unltyi 

The analysis of the alignment errors between points In 
two windows whose centers are Property aligned (liO, c=fsZ) 
will be presented for the special case of: 

S = Su ® Sv and a = P 

I I u'*U!V^»v I 1 12 

s < U*<S*cos(«)-i) + v*S*sln(a))t2 
+ (v»(S*cos(«>-l) - u*S*s!n<«Mt2 
5 ||U!V||t2 * C (1-S)t2 + 2*S*(l-oos(a) ) 3 

Hence we have a linear model! lie, 


1 1 U'-U! v' -V 1 1 

s Q(a,S) * 

1 1 U! V 1 1 

(B,4) 

Q(a,l) = 

2*sln<«/2> 


(B.5a) 

Q(0!S) » 

1 1 -s 1 


(B.Sb) 

If the centers of two 

correlation 

windows are 

proper | y 


(B.3a) 

(B.3b) 


aligned! then the misalignment error due to rotation and 
scale change at a Point n pixels from the center |s Q*n, 
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Thus far wt h&vs only estimated the masnltude of the 
error In the position of corresponding pixels based on our 
knowledge of the distortion process, We now want to know 
the effect of this misalignment on the value of the 
correlation function. 

FIsohler assumes (Flsehler C19713> that adjacent pixels 
are statistically lndoDendent« so that there Is no 
contribution to the product term ECX*Y> by multiplying 
Pixels which do not geometr I oa 1 1 y match. Flsehler then 
derives limits on the size of the correlation window as a 
function of the rotation and scale change. H|s derivation 
Is reinterpreted as follows* 

Assume that we have two arrays of pixels XCI#J3 and 
YCliJ? which are geometrically |n alignment, Suppose that 
we have a geometric distortion of the coordinate system such 
that a new array of pixels Z Is created, where aaoh pixel 
ZC1.J3 Is derived by bilinear Interpolation (the same 
derivation holds for higher order Interpolation and weighted 
averages In general) from a point at (u.v) In array Y, 

Let ksCuD. |aCv3) than 


ZU.J3 ■ a*Y[k.l3 + b*YEk+l»|3 * c*Y|:k,!+l] + d*YEk+l. i+1] (B,6) 


i ' 





where a.b.c. and d satisfy the conditions of eauatlons 
<4.5a-f ). 


Uet us assume for slmDiIcIty (with no real loss of 
qenerallty) that the Images have been normalized to zero 
mean Intensity va|ueS| F|sch|er's assumption of 
Indepenoenco than tmoMes that If |u-llSl or |v-J|>l then 
ZCt»J3 was derived from 4 points of Y which ape Independent 
of YCl*j3 and XCl»J3t Heneei such points contribute 
E(X)*E(Y3s0 to the sum of the XC ! p J3*Y|: I » J3 In equation 


{3.5) , 

If |u*t|<X and |v-J|<l then 

sum XC 1 p j]*2C I , JJ r sum aC I p J3»XC I r J3*YC I p J3 t8.7a) 

i.J l.J 

♦ <l«aC I p J] )*XC I p J3*(3 Independent points of Y) 

= sum anpj3*XCI,J3*YCt, j], (B,7b) 

I.J 

and there exists an A such that 0SA<1 and 

sum XCIpJ]*2Cl,j] s sum ( aC I p J ]»XC I p j 3»YC | , j 3 ) {B.8a) 

I p J I p J 

5 A * sum (XC I P JX*YC 1 1 J3) (B,0b) 

Ip J 

E(X«H) s A*ECX*Y) , <R,9) 


Therefore the only effect of the distorted coordinate 
system Is to attenuate the t(X«Y) term which attenuates the 
value of the correlation function. Of course, the value of 
A aepends on the magnitude of the geometric distortion 
errors of equation (0,3), 
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A Itinit . I'" -i of the oorrelation window Is 

derived fr.'ii*! by | I ml tine the maximum error to 

one pixel: 


» 1 I Uf V 


u'*u»v'**v|| / Q(tt|S} 




(B, 10 b) 


For Mariner 6 and 7 ploturesi the maximum combined 
rotation errors between two Images are about 2 degrees and 
the scale changes are less than about 2 %, Thereforei 


n »2 < l/(, 02 t 2 + 2 eat 02 >*a-cos( 2 ) ) ) 
or n < 25 


CB.U) 


FIsehler's Independence assumption essentially says 
that the autocorrelation functions for X and Y are one at 
u=0 and VS0 and zero elsewheroi We know that reasonable 
pictures do not have such autocorrelation surfaces, In 
partlculari the autocorrelation surfaces have central peaks 
whose width depends on the low frequency power of the power 
spectra of the Images, Consequently) knowing the 
autocorrelation function tells how large the errors In the 
alignment of two Images can be before the cross correlation 
Is adversely changed, If one has a choice of which areas In 
the two Images to a||gn» then the areas can be chosen on the 
basis of having broadjy peaked autocorrelation functions, 
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